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ABSTRACT 


The  principal  findings,  conclusions,  and  recommendations  of  the  five 
WSEIAC  Task  Groups  are  presented  in  summary  form.  The  system 
effectiveness  problem  is  examined  in  light  of  the  task  group  investigations. 
A  fifteen-step  procedure  for  cost-effectiveness  assurance  is  presented. 
Application  of  the  method  and  results  to  be  expected  in  each  phase  of  a 
system  life-cycle  are  described.  The  impact  on  existing  disciplines  is 
examined.  A  section  (Appendix  IV)  of  this  integrated  summary  contains 
abstracts  and  summaries  of  each  of  the  ten  reports  submitted  by  the  five 
Task  Groups.  Appendix  I  contains  a  more  detailed  treatment  of  the  fifteen 
recommended  tasks.  Appendix  II  presents  an  example  of  application  of  this 
methodology  shown  for  a  hypothetical  system  in  the  Conceptual  Phase. 
Finally,  Appendix  III  is  a  glossary  of  effectiveness/cost-effectiveness 
terms. 
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SECTION  I 


INTRODUCTION 

The  design  and  development  of  military  systems  has  traditionally 
crowded  the  state-of-the-art  in  materials,  devices,  and  technology.  In 
recent  times,  designers  have  been  faced  simultaneously  with  even  more 
novel  demands  and  acutely  limited  test  data.  Performance  requirements 
invariably  include  severe  reaction  times  which  can  be  met  only  by  closely 
integrating  personnel,  procedures,  and  hardware.  At  the  same  time,  pr  > 
grant  cost  limitations,  accelerated  development  schedules,  and  lack  of 
opportunity  for  complete  system  tests  prior  to  operational  deployment  hav 
reduced  the  opportunity  to  obtain  extensive  operational  usage  data.  Accord¬ 
ingly,  what  was  once  merely  considered  desirable  is  now  considered 

mandatory - an  integrated  methodology  of  system  management  using  all 

available  data  both  to  pinpoint  problem  areas  and  to  provide  a  numerical 
estimate  of  system  effectiveness  during  all  phases  of  the  system  life-cycle. 

A  general  recognition  of  this  situation  at  AFSC  headquarters  led  to  the 
formation  of  the  Weapon  System  Effectiveness  Industry  Advisory  Committee 
(WSEIAC),  Although  the  Air  Force  Systems  Command  recognized  the  need 
for  a  common  methodology  to  predict  and  measure  system  effectiveness, 
they  realized  that  the  problem  affected  too  many  other  organizations  to 
tackle  the  development  of  the  methodology  alone.  Thus,  they  submitted  a 
proposal  to  the  Secretary  of  the  Air  Force  to  create  a  committee  compost'd 
of  both  industry  and  Department  of  Defense  personnel.  The  committee  was 
formed  16  September  1963  by  the  AF  Systems  Command  with  the  approval 
of  Secretary  of  the  Air  Force,  Mr.  E.  Zuckert,  for  the  purpose  of  "prow 
ding  technical  guidance  and  assistance  to  the  Commander,  AFSC,  in  the 
development  of  a  technique  to  apprise  management  of  current  and  predicted 
system  effectiveness  at  all  phases  of  system  life." 

The  committee  was  composed  of  five  taBk  groups  of  approximately  ten 
members  each.  The  industry  members  were  "hired"  as  Bpecial  government 
employees  and  required  Secretary  of  Air  Force  approval  also.  They  ser  -d 
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without  compensation,  and  were  subject  to  rigid  conflict-of-interest  regula¬ 
tions.  From  the  Air  Force  there  were  members  of  the  Systems  Command, 
the  Logistics  Command,  the  Air  Training  Command,  the  Air  University, 
and  some  using  commands.  In  addition,  there  was  coordination  with  the 
Army,  Navy,  NASA  and  the  Department  of  Defense  through  participating 
observers . 

The  objectives  of  eaefi  task  group  were: 

Task  Group  I 

.  Review  present  procedures  for  establishing  system 
effectiveness  requirements. 

•  Recommend  a  method  for  determinging  system  effectiveness 
requirements  which  are  mission  responsive. 

Task  Group  II 

•  Review  existing  documentation  on  system  effectiveness. 

•  Recommend  methods  and  procedures  for  measurement  and 
prediction  of  system  effectiveness  in  all  phases  of  system  life. 

Task  Group  III 

.  Review  current  (Air  Force)  data  collection  and  reporting 
systems, 

•  Recommend  uniform  procedures  for  periodic  status 
reporting  to  assist  all  management  decision  levels. 

Task  Group  IV 

.  Develop  a  set  of  basic  Instructions  and  procedures  for 
conducting  analysis  for  system  optimization  considering: 

•  effectiveness 

•  cost 

<  program  time  scale. 

•  Refine  current  cost-effectiveness  analysis  techniques. 

Task  Group  V 

•  Develop  a  management  system  designed  to  absorb  and 
apply  system  effectiveness  experience  retention. 


Z 


Technical  reports  covering  each  task  group's  work  have  been  published. 
They  arc: 


AFSC-TR-65-1 

AFSC-TR-65-2 
( Vol.  I,  II,  III) 


Final  Report  of  Task  Group  1^ 
"Requirements  -  Methodology" 

Final  Report  of  Task  Group  11^’  ^ 

"Prediction  -  Measurement" 


AFSC-TR-65-3 

AFSC-TR-65-4 
(Vol.  I,  II,  III) 


Final  Report  of  Task  Group  III^ 

"Data  Collection  and  Management  Reports" 

Final  Report  of  Task  Group  IV^' 

"Cost -Effectiveness  Optimization" 


AFSC-TR-65-5 
(Vol.  I,  II) 


Final  Report  of  Task  Group  V(9,  10) 
"Management  Systems" 


The  Chairman's  Final  Report  is  a  synopsis  of  the  above  technical 
reports.  The  purpose  of  the  report  is  to  integrate  and  present  the  results 
of  the  WSEIAC  effort,  and  to  show  how  these  results  relate  to  Air  Force 
Systems  Management.  The  report  is  presented  in  two  volumes  ---  the 
present  Integrated  Summary  (Volume  II)  and  a  very  brief  report  which  is 
a  general  summary  (Volume  I). 
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SECTION  II 


WHAT  IS  THE  PROBLEM? 

The  accelerated  pace  of  design,  development  and  obsolescence  of 
military  systems  in  recent  years  has  given  rise  to  a  series  of  problems  in 
systems  management.  For  example: 

(1)  Many  of  today's  systems  have  a  high  unit  cost.  If,  in  the 
interest  of  economy,  the  national  budget  is  to  be  held  in  line  with  national 
objectives,  there  must  be  a  clear  indication  of  both  the  cost  and  effective¬ 
ness  of  a  proposed  system  long  before  the  decision  is  made  to  produce  the 
system  and  put  it  into  operational  use.  Thus,  there  is  a  clear  need  to 
predict  system  and  cost-effectiveness  as  early  in  the  system  life -cycle  as 
possible. 

High  unit  costs  also  lead  to  abbreviated  test  programs 
during  the  Acquisition  Phase.  Consequently,  there  is  a  large  degree  of 
uncertainty  in  the  quality  of  the  product  in  service  use.  Better  methods  of 
quantification  are  needed  to  reduce  this  uncertainty, 

(2)  Many  of  today's  weapon  systems  tend  to  be  "one  shot" 
devices.  As  a  result,  adequate  field  "debugging"  exercises  are  difficult  to 
accomplish.  There  is  less  direct  advance  evidence  as  to  the  adequacy  of  a 
system.  It  is  becoming  increasingly  necessary  to  rely  on  indirect  evidence 
for  assurance  of  effectiveness.  The  buyer,  usually  the  Air  Force  Systems 
Command,  must  have  a  way  to  assess  and  assure  the  ability  of  a  system  to 
meet  the  requirements  of  the  user,  usually  an  operational  command. 

(3)  Very  few  deny  the  necessity  of  defense.  Yet,  in  the  past 
few  years  there  has  been  ever  greater  emphasis  to  reduce  peacetime  de¬ 
fense  costs.  At  the  same  time  there  has  been  additional  emphasis  to 
increase  wartime  effectiveness.  Maximizing  effectiveness  and  minimizing 
costs  at  the  same  time  is  not  possible,  since  it  is  impossible  to  maximize 
and  minimize  two  dependent  variables  at  the  same  time.  Thus,  the  real 
problem  is  to  obtain  as  efficient  a  defense  posture  as  possible  within  the 
constraints  of  cost  and  effectiveness,  or  stated  another  way,  to  optimize 
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cost  when  the  effectiveness  is  constrained  or  to  optimize  effectiveness  when 
the  cost  is  constrained.  This  is  generally  referred  to  as  cost-effectiveness 
optimization.  Optimization  means  allocating  the  national  resources  in  a 
way  that  withstands  the  critical  vision  of  hindsight.  This  is  an  extremely 
difficult  problem  since  the  defense  posture  is  developed  in  the  presence  of 
risk  and  uncertainty.  Thus  it  is  essential  to  seek  and  use  the  best  avail¬ 
able  methods  for  cost-effectiveness  optimization, 

(4)  Many  programs  and  studies  have  been  conducted  under  the  name 
of  system  effectiveness,  operational  effectiveness,  operational  readiness, 
and  like  terms,  but  which  all  referred  to  the  same  problem.  A  need  existed 
to  gather  together  people  with  the  most  knowledge  in  this  problem  area  in 
order  to  standardize  definitions,  terminology,  and  a  method  of  attack, 

(5)  Finally,  there  is  the  problem  of  establishing  quantitative 
requirements  for  complex  systems,  particularly  when  those  requirements 
must  be  stated  in  probabilistic  terms.  The  severity  of  this  problem  may 
be  judged  from  the  following  WSEIAC  observation: 

The  minimum  acceptable  requirements  of  a  certain 
recent  SOR  are  given  piecemeal  in  terms  of  separate 
probabilities  and  performance  limits  without  obvious 
relation  one  to  another.  When  combined  in  an  over¬ 
all  system  effectiveness  number  (along  WSEIAC  lines) 
these  requirements  suggest  that  if  this  system  works 
less  than  4  times  out  of  100,  it  is  acceptable. 

These,  then,  are  some  of  the  problems  for  which  the  WSEIAC  sought 
solutions. 
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SECTION  III 


THE  WSEIAC  APPROACH  TO  THE  PROBLEM 

When  a  new  system  is  to  be  constructed  or  an  old  one  put  to  new  pur¬ 
poses,  there  are  two  diametrically  opposed  ways  of  proceeding: 

•  immediately  commit  resources  to  an  intuitively  plausible 

(re)design  and  surmount  the  problems  as  they  arise,  or 

■  explore  in  the  "mind's  eye"  the  consequences  of  the 

(proposed)  system  characteristics  in  relation  to  mission 
objectives  before  irrevocably  committing  resources  to 
any  specific,  approach. 

For  email  systems  and  simple  missions  either  way  may  prove  satis¬ 
factory,  and  indeed,  the  first  approach  can  be  the  superior  one  if  the  system 
designer  has  a  proven  record  of  successes.  On  the  other  hand,  painful 
experience  on  major  systems  has  shown  that  as  system  and  mission  com¬ 
plexity  increases,  a  point  is  reached  beyond  which  the  seat -of -the -pant a 
approach  simply  invites  economic  and  technical  disaster. 

It  is  not  surprising  therefore  that  the  WSEIAC,  in  its  recommendations, 
strongly  favors  the  second  approach.  In  particular,  emphasis  is  placed 
upon  methods  which  rely  heavily  upon  an  analytical  "model"  program.  In 
WSEIAC  usage,  a  model  is  any  device,  technique  or  process  by  means  of 
which  the  specific  relationships  of  a  set  of  quantifiable  system  characteris¬ 
tics  may  be  investigated.  For  example,  the  use  of  physical  scale  models  in 
marine  and  aeronautical  system  design  and  development  is  a  well  established 
and  sound  practice.  No  designer  of  ships  and  aircraft  would  dream  of  con¬ 
structing  a  prototype  (let  alone  entering  production)  until  he  has  tested  his 
ideas  in  a  tow  tank  or  wind  tunnel. 

In  the  past,  the  ultimate  user  has  continued  this  process  of  testing  and 
evaluation  by  subjecting  the  complete  system  to  special  "shakedown"  tests 
such  as  "war  games;"  that  is,  he  has  simulated  the  ultimate  mission  in  as 
realistic  a  manner  as  possible  without  actually  expending  the  system.  He 
has  "modeled"  a  tactical  situation. 
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In  more  re; cent  years,  as  mission  complexity  has  increased,  certain 
difficulties  have  arisen  which  make  this  logical  sequence  of  events  not  wholly 
satisfactory,  First,  the  use  of  scale  models  implies  that  the  range  of 
acceptable  performance  requirements  is  well  established  at  the  outset  of  the 
program,  and  it  is  only  necessary  to  cut  and  try,  with  the  aid  of  scale  models, 
to  meet  them.  The  difficulty  here  is  that  it  is  not  at  all  evident,  for  a  com¬ 
plex  mission,  just  what  an  acceptable  range  of  requirements  should  be, 
pa rtii  ularly  if  some  of  those  requirements  must  be  stated  in  probabilistic 
term  s . 

Second,  performance  of  full  scale  system  tests  under  realistic  condi¬ 
tions  io  becoming  impractical  for  several  reasons,  not  the  least  of  which  is 
the  increasing  cost  of  testing.  In  the  case  of  weapon  systems  and  certain 
space  projects  one  may  add  safety,  national  prestige,  political  considerations 
(such  as  nuclear  ban  treaties)  and  the  "one  shot"  nature  of  many  missions  as 
additional  constraints  on  testing.  Because  of  these  several  factors,  final 
committment  of  an  unproven  system  to  a  mission  is  an  increasingly  frequent 
occurrence. 

A  potential  solution  to  these  difficulties  is  the  judicious  use  of  analytic 
modeling  techniques  to  aid  both  in  establishing  subsystem  requirements  before 
development  commences,  and  to  compute  the  odds  for  mission  success  from 
less  than  full  system  test  data. 

In  effect  then,  one  is  forced  into  the  position  of  performing  an  analytic 
modeling  program  by  default.  Adequate  system  design  cannot  be  accom¬ 
plished  in  any  other  way. 

From  the  proceeding  considerations,  the  general  role  of  analytic 
modeling  is  clear.  Analytic  models  provide  insight.  They  make  an  empiri¬ 
cal  approach  to  Bystem  design  economically  feasible.  They  are  a  practical 
method  of  circumventing  a  variety  of  exterior  constraints. 

In  addition,  analytic  models  bring  to  bear  an  applicable  body  of  theorems 
on  stability,  asymptotic  behavior,  and  dynamic  performance. 

We  have  a  right,  then,  to  expect  certain  kinds  of  output  from  a  modeling 
program.  Clearly  a  modeling  program  should: 
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•  aid  in  establihing  requirements, 

•  provide  nn  asBessment  of  the  odds  for  flucressful 
mission  completion, 

•  isolate  problems  to  gross  areas, 

■  rank  problems  in  their  relative  seriousness  of  impact 
on  the  mission,  and 

•  provide  a  rational  basiB  for  evaluating  and  selecting 
between  proposed  system  configurations  and  proposed 
solutions  of  discovered  problems. 

Clearly  these  outputs  can  be  realized  only  if  the  scope  of  the  modeling 
effort  is  adequate  and  only  then  when  it  is  supported  by  a  reasonable  data 
base.  .  Furthermore,  these  outputs  are  achievable  only  when  the  words, 
"system  Effectiveness "  convey  a  definite  meaning  of  sufficient  scope, 

Thu  concept  of  system  effectiveness  has  been  expressed  many  times  in 
many  ways  by  many  people.  Sometimes  one  characteristic,  such  as  relia¬ 
bility,  has  been  emphasized  as  a  major  contributor  to  system  effectiveness. 
At  other  times,  other  characteristics  have  been  singled  out  for  special 
attention. 

The  time  has  come  to  concentrate  attention  on  the  primary  concern  of 
management  --  the  over-all  effectiveness  of  a  system  --  and  to  derive  a  way 
to  predict  and  measure  this  over-all  effectiveness  and  to  put  each  contri¬ 
buting  characteristic  in  its  proper  perspective  within  the  over -all  measure, 

A.  PRINCIPAL  FACTORS  OF  EFFECTIVENESS 

Thu  WSEIAC  has  taken  the  position  that  system  effectiveness  is  a 
quantitative  measure  of  the  extent  to  which  a  system  may  be  expected  to 
achieve  a  set  of  specific  mission  requirements.  It  is  expressed  as  a  func¬ 
tion  of  three  major  system  attributes: 

•  availability  (A) 

•  dependability  (D) 

•  capability  (C). 
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Availability  (A)  is  a  measure  of  the  condition  of  the  system  at  the 
start  of  a  mission,  when  the  mission  is  called  for  at  an  unknown  (random) 
point  in  time. 

Dependability  (D)  is  a  measure  of  the  system  condition  during  the  per¬ 
formance  of  the  mission;  given  its  condition  (availability)  at  the  start  of  the 
mission. 

Capability  (C)  is  a  measure  of  the  results  of  the  mission;  given  the 
condition  of  the  system  during  the  mission  (dependability). 

Cost-effectiveness  is  the  value  received  (effectiveness)  for  the  resources 
expended  (cost). 

You  will  note  that  the  WSEIAC  has  chosen  a  concept  and  definition  of 
effectiveness  based  only  on  quantifiable  factors.  There  are  certain  aspects 
of  the  problem  of  effectiveness,  and  an  effective  military  posture,  which 
are  purely  psychological.  An  effective  military  posture  1b  one  which  deters 
the  enemy;  or  given  that  this  doeB  not  occur,  will  abbreviate  the  conflict  in 
favor  of  our  national  interest, 

A  well  publicized  threat  of  missile  retaliation,  backed  in  actuality  by 
only  a  cleverly  concealed  squadron  of  "wooden  missiles,"  might  deter  the 
enemy  and  satisfy  the  first  half  of  the  above  requirement;  but  "wooden 
missiles"  would  not  satisfy  the  second  half  of  the  requirement.  However, 
it  is  difficult,  if  not  impossible,  to  quantify  or  assess  the  worth  or  value  of 
deterrence.  It  must  be  left  to  military  judgment.  Thus,  the  WSEIAC  con¬ 
cept  makes  no  attempt  to  quantify  these  psychological  factors.  However, 
Section  VII,  Volume  II  of  Task  Group  IV's  report,  "Risk  and  Uncertainty 
in  Cost-Effectiveness,"  discusses  the  differences  between  quantifiable  fac¬ 
tors  (which  are  called  risks)  and  non-quantifiable  factors  (called  uncer¬ 
tainties).^  ^  Risk  is  akin  to  rolling  dice  or  playing  roulette.  The  outcomes 
are,  on  the  average,  quantifiable  and  predictable.  Uncertainty  is  synonymous 
with  lack  of  information  or  inability  to  predict  the  outcome  of  the  future;  for 
example,  the  inability  to  prognosticate  future  weapon  system  configuration 
changes,  either  due  to  changes  in  hardware,  operational  concepts,  or  force 
size,  and  their  consequent  effect  on  costs.  Uncertainty  is  a  major  factor  in 
cost  overruns. 
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Thus,  oven  though  the  WSEIAC  definition  is  based  only  on  quantifiable 
factors,  the  differences  between  risk  and  uncertainty  and  between  quantifi¬ 
able  and  non-quantifiable  (such  as  psychological)  factors  were  recognized. 
Using  these  concepts  as  a  fundamental  point  of  reference,  current 
engineering  and  management  practices  were  examined. 

b.  effectiveness/cost-effectiveness  TASK  DESCRIPTIONS 

In  evaluating  the  work  of  the  task  groups,  a  logical  framework  for  con¬ 
ducting  System/Cost-Effectivenes s  Prediction  and  Analysis  has  evolved.  It 
is  reflected  in  the  systematic  fifteen  step  procedure  shown  in  Figure  1  (fold- 
out  on  page  21.  It  will  be  helpful  to  fold  the  figure  out  and  refer  to  it  now. 
This  flow  diagram  is  a  composite  representation  that  reflects  the  consensus 
of  the  five  WSEIAC  Task  Groups,  It  illustrates  the  sequence  (and  the  order) 
of  the  essential  tasks  that  must  be  performed  in  conducting  a  system  effec- 
tiveness/cost-effectiveness  prediction/evaluation/augmentation  cycle  in  the 
Conceptual,  Definition,  Acquisition  and  Operational  Phases;  i,  e. ,  to 
evaluate  (or  predict)  the  degree  of  effectiveness  that  has  (or  will)  be  attained 
for  any  achieved  (or  proposed)  system  configuration  and  to  augment  it  as 
required. 

The  choice  of  the  word  "prediction"  is  not  accidental.  It  is  impossible 
to  measure  effectiveness  short  of  total  war;  hence,  effectiveness  calculations 
always  contain  an  element  of  prediction.  The  object  of  these  predictions  is 
two  fold: 

•  SyBtem  effectiveness  predictions  form  a  basis  for  judging 
the  adequacy  of  our  defense  posture. 

•  Cost-effectiveness  predictions  form  a  rational  basis  for 
management  decisions. 

Efficient  use  of  such  predictions  will  provide  a  technical  key  to  more 
effective  selection,  definition,  development,  control,  evaluation  and  support 
of  a  system.  It  will  also  provide  a  basis  for  more  enlightened  program 
management, 

A  complete  cycle  is  defined  by  fifteen  essential  steps  commencing  with 
a  mission  definition  and  terminating  with  a  change  analysis.  Twelve  of 
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tlu:sc  steps  specifically  define  effectiveness  and  cost-prediction  activities; 
the  remaining  three  define  management,  system,  and  change  analysis 
activities. 

Since  a  large  portion  of  the  output  of  the  WSEIAC  is  devoted  to  presenting 
and  illustrating  the  use  of  these  steps  and  techniques  for  effectiveness/cost- 
effectiveness  prediction,  Appendix  I  has  been  added  to  discuss  each  step  in 
considerable  detail,  including  the  WSEIAC  recommendations  for  implemen¬ 
tation  of  each  step. 

These  steps  logically  fit  into  and  enhance  the  AFSC  375  series  manuals 
which  are  an  expression  of  current  Air  Force  program  management  philoso¬ 
phy  for  the  system  life -cycle.  The  activity  networks  for  each  phase  of  the 
system  life-cycle  shown  in  those  AFSC  375  series  manuals  contain  fund¬ 
amental  program  planning  and  surveillance  elements.  The  output  of  the 
WSEIAC  forms  an  integral  part  of  system  management.  Thus,  to  implement 
the  WSEIAC's  recommended  techniques,  the  activity  networks  of  the  AFSC 
375  series  of  manuals  must  be  revised  to  include  system  effectiveness 
critical  activities. 

It  should  be  carefully  noted  that  the  fifteen  steps  of  Figure  1  are 
intended  to  be  repeated  since  the  prediction  of  effectiveness /cost- 
offoctiveness  is  not  a  once-only  affair;  it  is  cyclic, 

.  In  the  Conceptual  and  Definition  Phases,  it  is  a  repetitive 
comparison  of  pairs  of  alternative  solutions  to  a  problem. 

•  During  the  Acquisition  Phase,  it  is  an  iterated  comparison 

of  current  status  with  requirements,  to  be  used  as  a  manage¬ 
ment  guide  in  the  allocation  of  resources. 

•  During  the  Operational  Phase,  it  is  both  a  periodic  monitor 
of  current  capability  and  a  tool  for  evaluation  of 
Bystem  improvements. 

In  spite  of  the  different  uses  cited,  effectiveness  and  cost-effectiveness 
predictions  may  be  conducted  by  a  similar  set  of  steps  in  each  phase  of  the 
system  life-cyclc.  The  next  section  explains  in  more  detail howthe  predictions 
and  the  WSLIAC's  findings  relate  to  each  phase;  hera  we  shall  only  briefly  indi¬ 
cate  the  nature  of  each  block  of  Figure  1.  For  further  detail,  see  Appendix  1, 
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BLOCK  1.0  MISSION  DEFINITION 


It  is  a  fundamental  requirement  of  the  methods  r ecoiruriended  by  the 
WSEIAC  that  a  clear  and  unambiguous  statement  of  the  mission  of  a  system 
bo  obtained.  This  definition  should  contain: 

•  a  description  of  the  purpose  of  the  system,  and 

•  system  quantitative  requirements. 

BLOCK  2.0  RESOURCES 

Resources  usually  evidence  themselves  as  a  practical  constraint  on  the 
development  and  procurement  of  a  system.  There  are  four  principal  areas 
of  consideration  here: 

•  budget 

.  persuiuu’l  resources 

•  industry  capacity 

•  technological  factors. 

An  example  of  the  way  in  which  technical  factors  are  specifically 
accounted  for  is  given  in  the  example  contained  in  Appendix  II  to  Volume  II 
of  the  Task  Group  IV  Final  Report.  ^ 

BLOCK  3.0  SYSTEM  DESCRIPTION 

System  description  consists  of  either: 

(1)  identification  of  alternative  system  configurations,  or 

(2)  configuration  documentation  \ 

followed  by 

(3)  a  system  summary  description. 

During  Ihe  Conceptual  Phase,  steps  (1)  and  (3)  form  a  logical  sequence. 
In  the  late  Definition  Phase  and  Acquisition  Phase,  the  emphasis  will 
increasingly  shift  to  steps  (2)  and  (3). 

The  object  of  the  last  step  is  to  present  an  uncluttered  picture  of  only 
thoBe  features  of  the  Bystem  structure  which  have  a  direct  bearing  on: 

the  estimation  of  system  effectiveness,  or 

•  a  cost -effectiveness  trade-off  Btudy. 
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BLOCK  4.  0 


FIGURES  OF  MERIT 


A  figure  of  merit  is  a  statement  which  relates  mission  (or  program) 
objectives  to  quantitative  system  requirements.  It  is  a  statement  of  the 
ability  of  a  system  to  meet  an  operational  need,  including  the  recognition  of 
the  risk  and  uncertainty  that  are  fundamental  characteristics  of  the  military 
mission. 

The  moat  comprehensive  figures  of  merit  have  been  dubbed  system 
effectiveness  and  cost-effectiveness.  System  effectiveness  is  a  quantitative 
measure  of  the  extent  to  which  a  system  may  be  expected  to  achieve  a  set  of 
specific  mission  requirements.  It  is  regarded  to  be  a  function  of: 

•  availability, 

•  dependability, 

•  capability. 

Cost-effectiveness  is  a  measure  of  the  value  received  (effectiveness)  foi 
the  resources  expended  (cost), 

BLOCK  5.0  SPECIFICATION  OF  ACCOUNTABLE  FACTORS 

As  a  preliminary  to  model  construction  and  following  mission  definitioi 
system  description,  and  specification  of  figures  of  merit,  it  is  necessary  to 
spoil  out  the  boundary  conditions  of  the  analysis  to  be  conducted.  First,  the 
level  of  accountability  must  be  specified: 

•  What  are  the  system  interfaces? 

•  What  is  the  depth  of  the  analysis  ? 

What  are  the  variables  of  the  analysis? 

Second,  it  is  necessary  to  define  constraints  on: 

•  data, 

•  schedule, 

•  burden, 

•  resources, 

•  acceptable  risk  and  uncertainty, 

•  physical  environment. 

In  addition,  it  is  necessary  to  spell  out  the  accountable  factors  in  the  areas  < 

•  personnel, 
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■  procedures, 

•  hardware, 

■  logistics. 

BLOCK  6.0  IDENTIFY  DATA  SOURCES 

The  detailed  structure  of  a  model  must  be  tailored  to  fit  the  type  of  data 
available.  This  is,  of  course,  a  two  way  road:  onlythoso  questions  may  be 
answered  for  which  data  exists.  Early  identification  of  data  sources  will  permit 
an  investigation  of  the  limitations  of  the  expected  data  sources  and  will  alert 
management  to  the  necessity  of  planning  to  acquire  supplementary  data. 

B LOG K  7 .  0  MODEL  CONSTRUCTION 

Model  construction  is  a  four  9tep  process: 

•  list  assumptions, 

•  list  variables  and  define  model  parameters, 

•  construct  effectiveness  model(s), 

<  construct  cost  model{s), 

The  listing  of  assumptions  is  crucial.  The  usefulness  of  a  model  can  be 
severely  restricted  if  the  assumptions  violate  reality.  A  clear  statement  of 
assumptions  is, therefore,  a  necessity  in  judging  the  validity  of  the  results  of 
a  model  exercise. 

Listing  variables  and  defining  the  model  parameters  permits  a  compari¬ 
son  of  the  structure  of  the  model  with  the  list  of  accountable  factors.  It 
provides  a  means  of  judging  the  completeness  of  the  model  structure. 

Effectiveness  models  should  reflect  the  three  major  system  attributes; 

•  availability, 

•  dependability, 

•  capability. 

Task  Group  II  has  given  several  explicit  illustrations  of  modeling  these 
factors  for  Air  Force  systems  in  Volume  II  and  Volume  III  of  their  final 
report. It  should  be  recognized  that,  although  all  but  one  of  the  illustra¬ 
tions  are  "pencil  and  paper"  models,  the  complexities  of  a  large  system  may 
very  well  require  that  the  actual  model  exercise  be  conducted  on  a  computer, 
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Task  Group  IV  presents  several  examples  of  cost-effectiveneBS  model 

(V  8) 

construction  in  Volume  II  and  Volume  III  of  their  final  report.'  *  ' 

BLOCK  8.  U  DATA  ACQUISITION 

Planning  for  data  acquisition  requires  careful  attention  to: 

•  specification  of  data  elements 

•  specification  of  test  methodology 

■  specification  of  a  data  collection  system. 

The  key  to  an  adequate  data  acquisition  program  is  the  determination  of 
those  system  events  which  are  significant.  A  system  event  is  only  of  signi¬ 
ficance  if  it  contributes  to  the  evaluation  of  a  parameter  of  the  system  model. 
Data  elements  are  only  significant  if  they  uniquely  locate  the  system  event 
in  space  and  time  with  respect  to  other  system  events. 

Frequently  it  is  necessary  to  answer  questions  which  call  for  special 
testing.  Maximum  utilization  of  the  acquired  data  can  be  achieved  only  if  the 
specification  of  test  methodology  is  accomplished  in  a  manner  responsive  to 
the  needs  of  the  system  model.  During  model  construction  any  special 
testing  that  may  be  required  should  be  communicated  to  those  responsible 
for  planning  for  data  collection. 

In  the  Air  Force,  specification  of  a  data  collection  system  requires  a 
consideration  of  "data"  in  a  broader  Bense  than  its  use  in  "data  element" 
above.  A  data  collection  system  is  the  organized  process  used  to  gather, 
store,  retrieve,  display,  publish,  and  distribute  a  wide  spectrum  of  system- 
related  information  including,  for  example,  training  manuals,  program 
plans,  management  summaries,  cost  data,  performance  data,  etc. 

BLOCK  9.0  DATA  PROCESSING 

The  processing  of  data  for  most  Air  Force  systems  is  a  large  under¬ 
taking  requiring  careful  attention  to: 

•  parameter  estimation  methods 

•  administrative  organization 

•  personnel  selection  and  training 

•  software  development 

•  hardware  specification  (computing  facilities), 
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The  specification  of  parameter  estimation  methods  is  a  crucial  step. 

The  scope  of  data  processing  is  so  large  that  it  is  unreasonable  to  assume 
that  those  who  process  data  are  aware  of  all  the  ramifications  of  their  work. 
Accordingly,  great  care  should  be  exercised  in: 

•  specification  of  effectiveness  parameter  estimation  methods,  and 

■  specification  of  cost  estimating  relationships. 

Techniques  in  these  latter  two  areas  are  discussed  by  Task  Group  II  and 
Task  Group  IV. ^ The  former  areas  are  treated  in  detail  by  Task 

(5) 

Group  III.  This  latter  task  group,  in  recognition  of  the  complexity  of  the 
data  acquisition  and  data  processing  tasks,  has  recommended  the  establish¬ 
ment  of  a  System  Information  Bank  (SIB)  for  each  Air  Force  system  and  a 
System  Effectiveness  Information  Central  (SEIC)  as  a  focal  point  for  system 
effectiveness  information  retention  on  an  Air  Force  wide  basis. 

BLOCK  10.0  SPECIFY  SCHEDULE 

Schedule  is  viewed  as  a  constraint.  It  is  assumed  that  schedule  control 
will  be  maintained  by  some  form  of  PERT.  In  addition,  schedule  should  be 
accounted  for  (possibly  implicitly)  in  the  system  effectiveness/cost- 
effectiveness  models.  '  ■ 

BLOCK  11.0  MODEL  EXERCISE 

There  are  two  principal  uses  of  models: 

•  evaluation  of  current  status, 

■  prediction  of  potential  status, 

Evaluation  provides: 

•  surveillance  o.'  cu..  r<- at  system  status  against  quantitative 
system  requirement.-, 

•  feedback  upon  live,  efficacy  of  the  management  decision 
process, 

•  a  means  of  determining  system  weaknesses  or  potential 
problem  areas, 
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•  a  point  estimate  of  system  effectiveness  which  includes  all 
relevant  factors  within  one  conceptual  framework. 

Prediction  provides  decision  aids  through  comparative  (cost-effective) 
analysis  of  competing: 

•  system  configurations,  and 
■  problem  solutions. 

The  use  of  a  system  model  involves  eight  steps: 

•  perform  checks  on  model, 

•  calculate  FOM's, 

•  do  trade-offs  within  constraints, 

.  compare  calculations  with  standard  of  reference, 

•  calculate  effect  of  risk, 

•  calculate  effect  of  uncertainty, 

y«  calculate  parameter  sensitivity  curves,  and 

•  interpret  runs. 

A  model  exercise  is  the  rational  basis  for  optimization  within 
constraints. 


BLOCK  12.0  PREPARE  MANAGEMENT  SUMMARY  REPORTS 


The  purpose  of  a  management  summary  report  is  to  communicate  the 
results  of  a  model  exercise  to  those  who  are  responsible  for  making 
decisions.  Hence,  It  must  be  executed  in  a  manner  that  aids  the  decision 
process.  The  management  summary  report  must  contain  not  only  the  main 
results  of  the  model  exercise  in  a  format  that  is  readily  understood,  but  in 
addition  it  should  contain: 


•  system  quantitative  requirements, 

•  current  system  status, 

•  resources  (remaining), 

•  trends, 

•  optimum  (re)allocation  of  resources,  and 

•  risk  and  uncertainty  qualifications. 
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BLOCK  13.0  DECISION  PROCESS 


Implementation  of  the  WSEIAC  recommendations  will  have  an  impact 
upon  the  decision  process.  Formal  effectivencss/cost-effectiveness  pre¬ 
diction  of  the  scope  envisioned  by  the  WSEIAC  is  without  precedent. 

Decision  processes  will  tend  to  become  more  formalized  and  prescribed. 

The  use  of  formal  decision  algorithms  will  become  more  widespread.  This 
does  not  mean  that  the  management  decision  process  has  been  relegated  to  a 
witless  computer.  It  does  mean  that  management  will  have  a  new  wealth  of 
correlated  facts  at  beck  and  call,  and  the  decision  process  will  become 
easier  and  more  accurate  in  many  instances.  Nevertheless,  the  ultimate 
act  of  decision  must  rest  with  a  human  who  can  account  for  the  qualitative 
aspects  of  the  world,  those  psychological  and  political  intangibles  that  the 
formalized  trade  studies  do  not  encompass. 

BLOCK  14.0  IMPLEMENT  DECISION 

It  should  be  carefully  noted  that  the  steps  of  the  task  analysis  under 
discussion  apply  in  any  and  all  phases  of  a  system  life-cycle,  Accordingly, 
implementation  of  a  decision  will  tend  to  differ  from  phase  to  phase.  In  the 
Conceptual  Phase  the  decision  may  take  any  of  the  following  forms: 

•  further  studies , 

•  initiation  of  research, 

•  initiation  of  exploratory  development , 

•  revision  of  an  existing  SOR  ,  or 

•  initiation  of  a  TSOR. 

In  the  Definition  and  later  phases,  decision  implementation  tends  to 
become  more  constrained.  For  typical  uses  of  the  WSEIAC  output  by  phase 
see  Section  IV. 

BLOCK  15.0  CHANGE  ANALYSIS 

The  implementation  of  a  decision  based  upon  effectivenes  s/cost-effectiveness 
considerations  generally  implies  a  change  in  one  or  more  of  thefollowing  areas: 

•  schedule , 

•  model(s), 

•  system,  nr 

•  requirements. 
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evaluation /augmentation  cycle  should  be  accompanied  by  a  change  analysis 
against  those  areas.  The  result  of  this  activity  will  be  a  monitoring  of  the 
net  effect  of  each  decision  and  the  accomplishment  of  program  surveillance. 


We  have  briefly  indicated  the  nature  of  each  task  in  Figure  1.  We  shall 
now  turn  to  a  brief  discussion  of  the  uses  of  the  WSEIAC  output  by  phase 
of  a  system  life-cycle. 
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FIGURE  1 


EXPANDED  ACTIVITY  I 
EFFECTIVENESS  PREL 
CYCLE  -  conceftua; 
OPERATIONAL  PHASE! 


c*n  occur. 


EXPANDED  ACTIVITY  NETWORK  OF  THE  STEPS  IN  A  SYSTEM 
EFFECTIVENESS  PREDICTION/EVALUATION/ AUGMENTATION 
CYCLE  -  CONCEPTUAL,  DEFINITION,  ACQUISITION  AND 
OPERATIONAL  PHASES 
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SECTION  IV 


USE  OF  WSEIAC  OUTPUT  BY  PHASE  OF  SYSTEM  .LJFE-OVCLE 


This  Section  presents  a  general  roadmap  illustrating  ttye  of  cost- 

effectiveness  prediction  techniques  in  each  phase  of  the  systVjljwi^e-cycle  as 

'  i 1  1 

defined  in  the  AFSC  375  series  of  manuals. 

A.  CONCEPTUAL  PHASE  / 

The  system  life  cycle  is  initiated  by  a  statement  of  a  general  need  for  a 
particular  operational  capability.  The  general  objective  of  this  phase  is  to 
establish  a  feasible  technical  approach  for  satisfying  the  general  require¬ 
ment,  to  evaluate  whether  a  specific  approach  is  worth  pur  suing,  or  whether 
the  military  requirement  should  be  satisfied  in  another  manner.  The  phase 
extends  from  determination  of  a  broad  objective  or  need,  to  Air  Force 
approval  of  the  Program  Change  Proposal  covering  the  Definition  Phase. 

The  Conceptual  Phase  studies  are  intended  to  serve  a  twofold  purpose: 

l 

•  provide  explicit  and  unambiguous  definitions  of  system 
effectiveness  for  the  particular  syBtem  to  which  the  Tentative 
Specific  Operational  Requirement  (TSOR)  applies; 

•  provide  guidelines  for  system  refinement  in  the 
Definition  Phase, 

The  WSEIAC  has  recommended  that  the  process  of  documenting  a  TSOR  or 
SOR  be  formalisted.  Toward  that  end  they  have  proposed  an  Air  Force 
manual  for  the  preparation  of  an  SOR  based  on  the  current  Headquarters 
Instruction,  HOI-DORQ  11-7.(1) 

In.  order  to  insure  that  the  requirements  of  the  SOR  are  met,  the 
WSEIAC  has  further  proposed  an  AFR  calling  for: 

•  the  pursuit  of  a  system  effectiveness  program  throughout 
the  life-cycle  of  a  system, 

•  the  establishment  of  a  System  Information  Bank  (SIB)  for  each 
system  during  the  Conceptual  Phase,  and 
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•  the  inclusion  of  Conc  eptual  Phase  studies  in  the:  TSOR  or  SOR 
(by  reference). 

Drafts  of  the  proposed  manual  and  regulation  arc  included  in  the  final 
report  ;f  Task  Group 

The  WSEIAC  also  recommends  that  quantitative  system  re.quirements 
should  not  be  made  absolute  and  firm  in  the  Conceptual  Phase.  The  output 
of  the  Conceptual  Phase  should  be  a  tentative  SOR  (TSOR).  Only  those 
requirements  rhich  can  be  fully  substantiated  as  being  absolutely  the  mini¬ 
mum  acceptable  should  be  made  firm  and  irrevocable.  Keeping  the  require¬ 
ments  as  flexible  as  possible  permits  better  trade-offs  in  later  phases, 
Figure  2  shows  a  simplified  activity  network  for  establishing  system  effec¬ 
tiveness  requirements  in  the  Conceptual  Phase. 

B.  DEFINITION  PHASE 

The  purpose  of  the  Definition  Phase  is  to  refine  the  definition  of  the 
system  to  the  subsystem  level  based  on  the  guidelines  established  in  the 
TSOR  and  by  the  Conceptual  Phase  studies. 

With  respect  to  the  simplified  activity  network  of  Figure  3,  the  inputs 
to  this  phase  are: 

•  definition  of  resources; 

•  definition  of  scenario; 

•  ■  definition  of  schedule  (time  scale); 

•  .  •  advanced  development  data; 

•  TSOR  containing: 

•  definition  of  effectiveness, 

•  preliminary  quantitative  requirements  , 

•  definition  of  primitive  system;  and 

•  Conceptual  Phase  studies. 

The  recommended  method  of  refinement  of  the  primitive  system  con¬ 
figuration  developed  in  the  Conceptual  Phase  is  the  iterated  performance  of 
the  following  sequence  of  steps: 

•  define  potential  cost-  effectiveness  improvement  (change)  to 
primitive  system; 
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potential  change; 


perform  a  cost-effectiveness  prediction  based  on  historical 
(generic)  data  and/or  advanced  development  data; 


refine  system  configuration  by  relative  cost-effectiveness 
ranking  of  alternative  improvements; 


iterate  steps  as  required. 

The  output  of  the  Definition  Phase  is  a  preliminary  use  plan  and  a  firm 
SOU  containing  at  least  the  following: 

■  the  dcfiii. .ion  of  system  effectiveness; 

■  minimum  acceptable  quantitative  system  requirements; 

•  documented  Conceptual  and  Definition  Phase  studies 
(by  reference). 

Additional  essential  outputs  are: 

•  subsystem  performance  specifications; 

•  cost  and  schedule  estimates. 

The  Definition  Phase  is  initiated  by  a  System  Definition  Directive,  and 
ends  with  issuance  of  a  System  Program  Directive. 

C,  ACQUISITION  PHASE 

The  purpose  of  the  Acquisition  Phase  is  to  develop  and  produce  the 
system  based  upon  a  firm  SOR. 

The  WSE1AC  recommends  that  during  this  phase,  cost-effectiveness 
techniques  be  used  as  a  management  aid  ini 

status  monitoring  of  system  development  against  the 
quantitative  requirements  of  the  SOR; 

■  allocation  of  resources. 


The  sequence  of  steps  which  must  be  performed  to  provide  the  relation 
among  these  factors  is  called  system  cost-effectiveness  prediction. 
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T'n!&  relationship  is  illustrated  in  .Figure  4,  which  specifically  indicates 
that  system  effectiveness/cost-effectiveness  prediction  is  the  focal  point 
which  provides  management  perspective  as  to  the  relationship  between 
system  status,  available  resources,  constraints,  and  system  requirements. 

This  phase  starts  after  issuance  of  the  System  Program  Directive  and 
ends  with  acceptance  by  the  user  of  the  last  operating  unit  in  a  certain  series, 
or  until  the  SOR  has  been  demonstrated  through  Category  II  testing  and  all 
required  updating  changes  resulting  from  the  testing  have  been  identified, 
approved,  and  placed  on  procurement,  whichever  occurs  later. 

D .  OPERATIONAL  PHASE 

The  Operational  Phase  is  identified  as  the  period  of  system  use  (and 
field  support).  During  this  phase  of  system  life,  the  techniques  recom¬ 
mended  by  the  WSEIAC  will: 

•  aid  the  using  command  in  performing  a  critical  evaluation 
of  the  system; 

•  aid  the  support  command  In  achieving  an  economical  and 
timely  support  of  the  system  by  verifying  the  earlier  deter¬ 
mined  provisioning  and  in  evaluating  proposed  modifications. 

Figure  5  indicates  the  relationship  between  system  effectiveness/cost- 
effectiveness  prediction,  the  System  Information  Bank  (SIB),  the  using/ 
support  management,  and  system  activities. 

As  in  the  Acquisition  Phase,  system  effectiveness/cost-effectiveness 
prediction  is  the  focal  point  which  provides  management  perspective. 

This  phase  begins  with  acceptance  by  the  user  of  the  first  operating 
unit,  continues  until  final  disposition  of  the  system. 
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FIGURE  4 

SIMPLIFIED  ACTIVITY  NETWORK  FOR  SYSTEM  REFINEMENT 
AND  STATUS  MONITORING  DURING  THE  ACQUISITION  PHASE 
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OPERATIONAL.  PHASE 


FIGURE  5 

SIMPLIFIED  ACTIVITY  NETWORK  FOR  OPERATIONAL 
EVALUATION  AND  SUPPORT  DURING  OPERATIONAL  PHASE 
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SECTION  V 


EFFECTIVENESS  ASSURANCE  MANAGEMENT 

A.  INTRODUCTION 

System  effectiveness  has  two  major  aspects.  The  first  is  to  provide  a 
quantitative  basis  for  establishing  requirements  during  project  inception  and 
definition  and  for  evaluating  achievements  during  acquisition  and  operation. 
The  second  is  to  provide  management  disciplines  that  will  allow  achievement 
of  the  predicted  optimum  levels  of  system  effectiveness.  To  do  so  requires 
resources  development  for  each  activity  that  is  critical  to  system  effective¬ 
ness  and  development  of  program  management  methods  to  assure  effective 
application  of  these  resources, 

It  is  not  enough  to  predict  what  could  be  achieved  by  good  engineering, 
good  manufacturing  and  good  management  and  to  be  able  to  measure  later 
achievements.  It  is  necessary  to  develop  and  apply  techniques  that  will 
assure  that  engineering,  manufacturing  and  management  are  performed  in  a 
way  that  will  realize  the  potential  system  effectiveness.  The  Air  Force  has 
provided  the  basis  for  such  assurance  through  program  management  techno¬ 
logy  in  accordance  with  the  AF  375  series  of  documents.  Unfortunately, 
these  documents  do  not  yet  clearly  identify  all  the  steps  that  must  be  taken 
to  provide  system  effectiveness  assurance;  neither  do  they  deal  with 
resources  development  in  the  form  of  the  technology  and  people  trained  and 
motivated  specifically  in  system  effectiveness  techniques. 

B.  THE  MANAGEMENT  CONCEPT 

Task  Group  V  has  identified  six  major  segments  of  management  inherent 
in  the  development  and  application  of  resources  toward  achievement  of 
system  effectiveness.  These  segments  fall  into  two  groups.  The  first, 
Resource  Development  (Experience  Retention),  includes: 

Data  Acquisition 
Techniques  Development 

•  Personnel  Development. 


The  second  group.  Resource  Application  (Program  Management),  includes: 

•  Program  Planning 

•  Input  Surveillance 

•  Output  Evaluation. 

The  recognition  and  disciplined  treatment  of  activities  critical  to 
effectiveness  included  within  each  of  these  six  segments  constitute  the 
management  concept.  Critical  activities  are  those  activities  that  experience 
has  shown  must  be  subjected  to  formal  discipline  in  order  to  assure  system 
effectiveness . 

Within  this  concept,  "discipline"  is  equated  with  all  types  of  control 
over  the  activities  of  people.  It  consists  of  training,  motivation,  command, 
and  audit.  Application  of  these  principles  to  specifically  identified  system 
effectiveness  critical  activities  allows  evaluation  of  current  status  and 
defines  those  areas  where  specific  improvement  can  be  introduced.  For 
example,  an  Air  Force  program  director  may  receive  general  training  in 
program  management  philosophy  but  still  be  left  in  doubt  about  specific 
actions  that  he  should  take  in  managing  a  new  program,  By  contrast,  if  he 
is  taught  that  there  is  a  tangible  activity  called  "functional  flow  analysis,  " 

--  if  he  is  provided  with  documented  technology  for  this  activity  and  moti¬ 
vated  to  apply  it,  --  if  he  is  commanded  by  AFSCM  375-5  to  require  his 
contractors  to  schedule  and  fund  the  performance  of  this  activity,  --  and  If 
the  program  is  audited  by  his  inspector  general,  there  is  a  high  probability 
that  the  activity  will  be  accomplished, 

C.  THE  SIX  SEGMENTS  OF  ASSURANCE  MANAGEMENT 

1.  Data  Acquisition  In  general,  data  acquisition  systems  are  estab¬ 
lished  to  provide  program  directors  with  information  for  the  management  of 
the  project  from  which  the  data  is  obtained.  In  fact,  AFSC/AFLC  310-1 
(Management  of  Contractor  Data  and  Reports)  restricts  data  acquisition  to 
this  purpose.  For  resources  development  purposes,  it  is  necessary  to 
develop  data  feedback  into  forms  suitable  for  decision  assurance  on  future 
projects.  For  example,  data  on  the  generic  failure  rates  of  electronic  parts 
must  be  obtained  from  current  projects  and  fed  forward  for  use  in  predicting 
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the  reliability  of  future  systems,  Resource  development  date  requirements 
include  engineering  type  data  obtained  by  dissection  and  analysis  of  failed 
parts.  However,  information  on  management  lessons  learned  in  fields  such 
as  source  selection,  cost  estimation,  and  manufacturing  process  control 
also  must  be  acquired  and  fed  back.  Data  requirements  are  shown  as  a 
portion  of  the  input  prior  to  Block  1.  0  and  in  Blocks  6,0,  8.0  and  9.  0  of 
Figure  1  (page  21). 

2.  Technology  Development  Many  important  techniques,  such  as 
electronic  stress  analysis,  design  review,  or  production  environmental 
testing,  start  out  by  competent  people  actually  performing  them  as  required 
to  support  a  particular  project.  Temporary  technical  excellence  may  be 
developed  by  such  effort,  but  this  is  not  enough.  In  order  to  assure  repeti¬ 
tion  of  successes  and  avoidance  of  repetition  of  failures,  and  in  order  to 
spread  technical  excellence  throughout  the  industry,  It  is  necessary  to 
document  the  technology.  Such  documentation  covers  lessons  learned  on 
all  projects  and  thereby  represents  the  best  of  which  industry  is  capable. 
Although  the  fifteen-step  procedure  outlined  in  Section  III  provides  a  step-by- 
step  framework  against  which  to  perform  an  effectiveness/cost-effectiveness 
analysis,  considerable  development  of  the  necessary  techniques  is  needed. 
These  are  outlined  in  Section  VI,  "General  Conclusions  and  Recommenda¬ 
tions." 

3.  Personnel  Development  Documented  technology  does  not  of  itself 
achieve  results.  People  must  be  taught  the  technology  and  motivated  to  use 
it.  Such  people  then  constitute  the  primary  resource  for  successful  execu¬ 
tion  of  new  projects.  Task  Group  V  has  identified  training  needs  and  called 
for  additional  programs  within  the  Air  University  and  the  Air  Training 
Command.  Similar  effort  is  required  within  industry. 

4.  program  Planning  Each  of  the  Air  Force  375  series  documents 
includes  an  activity  network.  These  may  be  regarded  as  model  program 
plans.  They  identify  activities  that  must  be  required  and  funded  by  the 
System  Program  Office.  In  addition,  they  identify  two  types  of  output  from 
these  activities.  The  final  output  from  decision  making  disciplines  may  be 
called  "decision  disclosure  documents."  For  example,  a  specific  operational 


37 


requirement  is  a  disclosure  of  decisions  made  during  the  Conceptual  Phase 
of  a  system  life-cycle.  These  decision  disclosure  documents  are  like  the 
baton  in  a  relay  race.  They  are  the  tangible  item  that  is  passed  on  from 
one  group  to  the  next  or  from  one  phase  to  the  next.  The  second  type  of 
output  may  be  called  "decision  guide  data."  Such  data  is  generated  by  the 
activities  as  a  basis  for  the  decisions  set  forth  in  decision  disclosure  docu¬ 
ments.  For  example,  the  activity  known  as  "functional  flow  analysis" 
results  in  "functional  flow  block  diagrams."  This  data  is  essential  to 
assuring  the  quality  of  decisions  made  by  the  systems  engineer,  and  these 
decisions  are  reflected  in  "end  item  detail  specifications." 

Although  AF  375  series  networks  are,  in  fact,  model  program 
plans,  actual  plans  are  prepared  in  many  forms,  Some  are  narrative,  some 
tabular,  and  some  are  combinations  of  these  two  basic  forms.  In  all  cases, 
program  plans  must  show  which  system  effectiveness  critical  activities  are 
required  and  funded.  In  addition,  they  should  show  who  is  responsible  for 
their  execution,  when  they  will  be  performed,  what  decision  guide  data  will 
be  generated,  and  which  decision  disclosure  documents  will  be  affected  by 
this  data. 

5.  Input  Surveillance  The  ideal  relationship  between  a  buyer  and  a 
seller  is  one  in  which  the  interface  is  limited  to  the  buyer  specifying  exactly 
what  he  wants  and  providing  qualification  tests  and  receiving  inspection 
criteria  to  assure  that  he  receives  it.  This  type  of  relationship  may  be 
described  by  the  term  "output  contracting."  For  this  relationship  to  be 
satisfactory,  it  is  essential  that: 

•  the  buyer  can  specify  numerically  every  requirement  for 
the  product,  including  reliability  and  maintainability  values; 

the  buyer  and  seller  can  agree  on  a  funded  demonstration  test 
that  will  prove  that  all  numerical  requirements  have  been  met; 

the  buyer  can  tolerate  the  schedule  delay  and  extra  cost  that 
will  result  if  the  seller  fails  to  pass  a  specified  demonstration 
test. 

Unfortunately,  it  is  seldom  possible  to  negotiate  and  fund  an 


demonstration  plan  for  reliability,  maintainability,  and  all  the 
other  characteristics  which  contribute  to  system  effectiveness.  It  is  even 
more  rare  for  a  buyer  to  be  in  a  position  to  tolerate  the  schedule  slippage 
and  extra  cost  that  would  occur  if  the  contractor  failed  to  meet  specified 
requirements.  Consequently,  the  Air  Force  has  found  it  necessary  to  supple¬ 
ment  output  contracting  by  specifying  that  activities  critical  to  program 
objectives  will  be  performed  by  their  contractors  in  accordance  with  disci¬ 
plined  procedures,  This  may  be  called  "input  contracting." 

It  follows  that  contract  management  practices  have  had  to  be 
extended  to  include  surveillance  over  the  accomplishment  of  critical  activi¬ 
ties.  This  surveillance  may  take  the  form  of  assignment  of  engineering 
development  officers  as  Air  Force  plant  representatives  and  as  chairmen 
of  technical-direction  meetings  at  the  contractor's  plant.  All  of  these 
practices  may  be  described  by  the  term  "input  surveillance." 

6.  Output  Evaluation  Program  management  technology  predicts  that 
if  work  statements  and  program  plans  are  followed,  certain  results  will  be 
obtained.  It  is  necessary  to  assure  that  actual  results  are  as  predicted,  and, 
if  they  are  not,  to  take  rapid  corrective  action.  The  techniques  for  doing  so 
may  be  described  by  the  term  "output  evaluation."  This  segment  of  the 
system  effectiveness  assurance  management  system  includes  design  reviews, 
qualification  testing,  receiving  inspection,  and  Category  I,  II,  and  III 
system  testing.  It  also  includes  failure  analysis,  feedback  and  corrective 
action  by  project  personnel,  and  effectivoness/cost-effectiveness  evaluation. 

Figure  6  illustrates  the  relationship  of  the  above  six  segments  of 
effectiveness  assurance  management  as  a  closed  loop  process.  The  left 
hand  arrow  illustrates  that  operational  executives  are  responsible  for 
obtaining  resources  from  functional  groups  and  applying  them  to  each  critical 
activity.  The  right  hand  arrow  illustrates  that  functional  executives  must 
acquire  and  document  the  experience  of  present  and  past  programs  as  an 
essential  basis  for  new  programs.  Development  of  technical  excellence  for 
each  critical  activity  in  a  form  that  can  be  transferred  to  new  projects 
depends  on  this  continuous  feedback  from  current  projects. 
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D.  CRITICAL  ACTIVITIES 


While  it  may  be  said  that  nearly  all  aspects  of  a  program  are  critical 
to  system  effectiveness  in  the  broad  sense,  the  WSEIAC  has  identified  a 
technical  framework  for  relating  the  many  system  characteristics  through 
the  terms  availability,  dependability,  and  capability. 

Each  of  these  major  system  attributes  and  their  principal  factors  must 
be  identified,  measured  and  controlled  to  acceptable  limits  during  the  pro¬ 
gram  development  cycle.  Some  examples  of  critical  activities  associated 
with  system  effectiveness  characteristics  are: 

Functional  Flow  Analysis 
Failure  Mode  Determination 

•  System  Safety  Analysis 
Quality  Assurance 

•  Reliability  Prediction 
Maintainability  Analysis. 

The  WSEIAC  has  not  emphasized  the  many  detailed  considerations 
within  the  above  activities.  Rather,  the  attempt  has  been  to  pull  them 
together  as  a  cohesive  whole. 

The  fifteen-step  procedure  described  in  Section  III  is  an  attempt  to 
relate  all  of  these  critical  activities  in  a  logical  way  to  permit  an  evaluation 
of  the  composite  figure  of  merit  during  successive  stages  of  a  program 
life  -cycle . 

E.  CONCLUSION 

It  is  a  predominant  conclusion  of  the  WSEIAC  that  the  major  segments 
of  effectiveness  management  described  above,  and  the  specific  critical 
activities  implied,  exist  today  in  a  piecemeal  fashion  only,  or  are  not  pre¬ 
sent  at  all.  The  policy  issues  identified  by  Task  Group  V  and  the  recom¬ 
mendations  in  this  integrated  report  are  directed  toward  major  improvement 
of  this  situation. 

In  summary,  the  WSEIAC  recommends  the  introduction  of  a  degree  of 
formalization  into  the  management  process.  This  can  be  done  by  integration 
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of  the  WSEIAC  procedures  into  the  AF  375  series  documents  so  that 
evaluation  and  prediction  of  effectiveness/cost-effectiveness  is  in  proper 
relation  to  the  various  assurance  activity  networks  of  the  series. 


SECTION  VT 

GENERAL  CONCLUSIONS  AND  RECOMMENDATIONS 

Wc  have  briefly  discussed  the  basic  problems  which  led  to  the  formation 
of  the  WSEIAC  and  the  approach  to  their  solution.  These  problems  were 
broadly  categorized  as  "system  effectiveness"  problems.  Their  solution 
calls  for  a  knitting  together  of  a  wide  spectrum  of  existing  management 
disciplines  and  a  variety  of  technical  specialties. 

Problems  have  been  identified  and  recommendations  for  their  solution 
have  been  given.  The  proposed  solution  is  a  system  effectiveness  assurance 
management  program.  The  fundamental  requirement  of  this  program  is  that 
program  management  utilize  a  formalized  structure  of  trade-off  analyses 
based  upon  the  principles  of  cost-effectiveness  optimization.  The  principal 
steps  of  that  structure  were  shown  in  Figure  1.  A  large  fraction  of  the  out¬ 
put  of  the  task  groups  was  devoted  to  spelling  out  the  tasks  in  this  formalized 
structure  and  are  discussed  in  detail  in  Appendix  I.  Significant  findings  of 
the  five  task  groups  are  given  in  the  following  paragraphs. 

A.  data  ACQUISITION 

The  requirement  for  data  is  so  generally  acknowledged  as  fundamental 
to  any  decision  process  that  almost  any  remark  concerning  data  has  a  ten¬ 
dency  to  sound  trite,  In  spite  of  this,  the  WSEIAC  survey  of  current  data 
collection  systems  failed  to  discover  a  single  one  that  was,  of  itself,  capable 
of  providing  satisfactory  system  effectiveness  data. 

There  are  a  variety  of  reasons  for  this  unhappy  situation: 

•  Data  for  effectiveness  prediction/evaluation  must  be  acquired 
by  scientific  planning.  A  proper,  definitive  list  of  the  minimum 
data  elements  required  to  evaluate  a  system  must  be  based  on 
the  needs  of  a  system  model,  not  generated  haphazardly  to 
satisfy  specialized  needs,  as  is  the  current  practice. 

•  Current  Air  Force  data  collection  systems  are  inflexible. 

They  are  not  responsive  to  changing  requirements. 
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Current  Air  Force  data  reporting  systems  are  Bubject  to  a 
variety  of  correctable,  but  currently  uncorrected  error 
sources. 

•  Current  Air  Force  reporting  practice  is  to  report -by -exception. 
Successful  event  information  is  largely  missing. 

■  Data  on  a  given  system  is  not  centralized.  It  may  never 
leave  a  contractor's  plant  or  the  base  at  which  it  is  generated. 

On  the  other  hand,  that  data  which  is  disseminated  is  sent 
piecemeal  to  a  variety  of  locations.  No  one  agency  receives 

a  complete  data  package. 

■  Historical  data  does  not  exist  in  general,  Current  security 
practices  and  the  cost  of  storage  encourage  the  destruction 
of  data  after  short  periods  of  retention.  For  example,  a 
survey  was  conducted  on  the  availability  of  Atlas  D  ICBM 
development  data.  Virtually  none  was  discovered  although 
this  weapon  system  has  been  obsolete  for  about  only  one 
year. 

The  recommended  solution  to  these  problems  is  the  establishment  of  a 
centralized  System  Information  Bank  (SIB)  for  each  system.  The  SIB  would 
be  maintained  throughout  the  life  of  the  system.  During  the  Definition  Phase 
and  Acquisition  Phase,  the  SIB  would  be  established  and  maintained  by  the 
AFSC/SPO.  During  the  Operational  Phase,  the  SIB  would  be  maintained  by 
the  Logistics  Command  and  be  supported  by  the  using  command.  A  sum¬ 
marized  version  of  system  data  would  be  forwarded,  on  demand,  to  a 
System  Effectiveness  Information  Central  (SEIC),  where  it  would  be  available 
to  all  on  a  need-to-know  basis.  It  is  recommended  that  the  initial  implemen¬ 
tation  of  SEIC  be  under  AFSC  sponsorship. 

A  proposed  charter  and  organization  of  SEIC  is  given  in  the  final  report 

of  Task  Group  V.  Proposed  methods  of  operation  of  SEIC  are  described 

(51 

in  the  final  report  of  Task  Group  III. '  ' 
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B .  TEC HN IQUE  DEVELOPMENT 

A  large  portion  of  the  WSEIAC  report  is  devoted  to  illustrating  the 
applicability  of  current  techniques  in  predicting/evaluating  system  effective- 
noss/cost-effcctiveness.  However,  these  reports  are  not  a  complete  set  of 
"how-to"  manuals. 

Several  problem  areas  requiring  further  definitive  treatment  are  cited: 

•  There  are  no  standardized  techniques  for  the  establishment 
of  minimum  acceptable  quantitative  system  requirements, 
particularly  when  those  requirements  are  stated  as 
probabilities . 

There  is  no  definitive  list  of  basic  raw  data  elements  for 
any  extant  system,  simply  because  an  effectiveness/cost- 
effectiveness  assurance  program  of  the  scope  envisioned  by 
the  WSEIAC  is  unprecedented. 

■  Because  both  system  effectiveness  and  cost-effectiveness 
calculations  are  in  their  infancy,  there  are  no  standardized 
techniques  for  effectiveness  prediction/evaluation/demonstration. 

The  use  of  models  in  management  decision  processes  needs 
considerable  clarification.  There  is  a  suspicious  amount 
of  evidence  that  current  use  of  models  falls  into  either  of 
two  categories: 

*  If  the  model  output  agrees  with  a  preconceived 
position  ---  use  it  as  a  vindication  of  judgment. 

If  the  model  output  disagrees  with  a  preconceived 
position  or  an  accomplished  fact  ---  suppress  it. 

The  underlying  reasons  for  this  deplorable  situation  are  two  fold: 

untimcliness  of  model  output,  and 
lack  of  confidence  in  model  output, 

Complex  systems  tend  to  lead  to  complex  models.  Complex  models  end 
up  as  large  scale  computer  simulations  which  take  time  to  construct  and 
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"debug,"  ft i til  which  lake  additional  limit  to  reconstruct  as  the  system  changes. 
Tims  a  timely  output  in  response  to  a  management  query  becomes  difficult. 

Confidence  in  the  model  output  is  closely  related  to  the  validity,  accura¬ 
cy  and  quantity  of  the  input  data.  The  inadequacy  of  current  planning  for 
acquiring  data  lends  to  reduce  confidence  in  model  outputs. 

'  The  methods  of  estimation  of  effectiveness  parameters 
needs  development.  The  one-shot  nature  of  many  current 
systems,  combined  with  high  unit  costs,  lead  to  small 
test  sample  sizes.  The  need  for  methods  of  nondestructive 
testing  and  statistical  inference  techniques  is  clearly 
indicated. 

•  Methods  of  system  cost-effectiveness  optimization  need 
further  development,  Current  optimization  techniques 
are  not  wholly  adequate  to  the  task  of  manipulating  a 
large  number  of  variables  which  may  be  non-analytical 
and  frequently  discontinuous  except  for  short  intervals. 

•  Methods  of  incorporating  risk  and  uncertainty  into  models 
need  development.  Placing  confidence  levels  (risk)  on 
the  output  of  models  is  an  unsolved  problem. 

The  WSELA.C  recommendations  here,  are  several  fold: 

•  The  problem  of  establishing  meaningful  requirements  should 
be  submitted  to  further  study  by  one  or  more  competent 
agencies. 

•  A  task  group  should  be  formed  for  the  specific  purpose  of 
defining  a  m  nimum  list  of  basic  effectiveness /cost-effec - 
tiveness  raw  data  elements  on  each  of  the  several  larger 

AF  systems.  This  list  should  be  directly  reflected  in  changes 
in  current  data  collection  forms  and  practices. 

•  The  techniques  for  effectiveness /cost -effectiveness  pre¬ 
diction  provided  by  the  WSEIAC  should  be  regarded  only  as 

a  point  of  departure.  It  is  recommended  that  these  techniques 
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be  expanded  and  published  in  manual  form. 

•  The  entire  question  of  the  use  of  mathematical  models  in  the 
management  decision  process  should  be  subjected  to  a  care¬ 
ful,  definitive  study  with  particular  emphasis  on  (1)  methods 
of  providing  timely  outputs  from  complex  models,  (2)  prob¬ 
lems  associated  with  the  use  of  small  data  sample  size,  (3) 

the  refinement  of  optimization  techniques,  and  (4)  the  boundaries 
within  which  intelligent  decisions  may  be  made. 

C.  PERSONNEL  DEVELOPMENT 

Application  of  the  techniques  recommended  by  the  WSEIAC  calls  for 
a  continuing  personnel  development  program  in  the  Air  Force  and  in  industry. 
Several  Air  Force  problem  areas  have  come  to  light  here: 

•  SPO  manning  in  the  past  has  lagged  contracts  so  that 
adequate  program  control  has  not  always  been  available 
when  required, 

•  The  personnel  system  of  the  Air  Force  does  not  adequately 
recognize  the  experience  retention  aspects  of  personnel 
development. 

•  The  personnel  system  of  the  Air  Force  does  not  adequately 
reflect  the  functional  aspects  of  Air  Force  requirements. 

•  Experience  gained  on  any  given  system  is  inadequately 
reflected,  or  is  not  reflected  at  all,  in  new  system 
development. 

Most  of  the  difficulties  cited  above  can  be  mitigated  by  greater  utiliza¬ 
tion  of  existing  technical  staffs  for  development  and  review  of  technical 
requirements,  work  statements,  RFQ's,  etc,  during  the  Conceptual  and 
Definition  Phases  when  the  need  is  most  critical, 

In  addition,  however,  existing  Air  Force  programs  for  the  acquisition, 
training,  and  motivation  of  its  management  and  technical  staffs  require  con¬ 
tinued  firm  support.  A  positive  program  ’  .eeded  to  identify  personnel 
skills  and  experience,  and  then  to  match  these  with  open  requirements.  The 
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development  of  a  system  for  retention  of  experienced  personnel  through 

sound  personnel  policies  and  good  communication  within  management 

,  „  (9,  10) 

structures  appears  to  need  increased  effort. 

D.  PROGRAM  SURVEILLANCE 

Admittedly,  it  would  be  desireable  for  a  user  or  customer  to  specify 
exactly  what  he  wants  in  terms  of  end  performance  objectives  and  to  call 
for  such  demonstration  prior  to  accepting  delivery  and  issuing  pavment. 
There  are  many  instances  within  the  government/industry  procurement 
program  whore  this  is  indeed  possible.  However,  for  many  of  our  complex 
systems  which  extend  the  state-of-the-art  across  a  broad  spectrum  of 
technology,  this  simplified  approach  is  just  not  feasible. 

Development  time  scales,  fiscal  limitations,  technical  unknowns,  and 
other  constraints  force  the  procurement  into  a  sequential  process  where 
detailed  milestones,  interim  objectives,  and  even  methods  to  be  used  in 
arriving  at  the  distant  end  objectives,  are  specified  and  controlled.  Pro¬ 
gram  surveillance  by  the  customer  has  become  a  way  of  life;  and  the:  seller 
is  committed  to  a  continuous  demonstration  that  his  process  for  attaining 
the  end  objectives  is  in  control. 

There  is  no  sure  cure  for  these  problems.  However,  surveillance  by 
staff  officers  during  the  preparation  of  work  statements  and  requirements 
during  the  Conceptual  and  Definition  Phases,  assignment  of  engineering 
development  officers  as  Air  Force  PI  x  Representatives,  holding  technical 
direction  meetings  among  associate  contractors,  and  incorporation  of 
system  effectiveness /cost-effectiveness  principles  in  the  AFSC  357  series 
of  manuals  --  all  will  tend  to  improve  surveillance. 

E.  GENERAL  RECOMMENDATIONS 


It  is  not  the  intent  of  this  summary  to  present  detailed  recommendations 


since  they  aVtycoverod  in  each  task  group’s  report,  but  rather  to  indicate 
briefly  those  principal  steps  which  can  and  should  be  taken  toward  imple¬ 
menting  the  WSEI,AC  results  as  early  as  possible.  These  steps  are: 

Establish  a  responsible  office  i(headed  by  a  general  officer)  at 
,  Air  Staff  level  and  supporting  offices  at  all  appropriate  sub¬ 
ordinate  levels  for  the  purpose  of  initiating,  coordinating, 
monitoring,  and  implementing  a  system  effectiveness  program 
within  the  Air  Force  and  for  coordinating  this  program  with 
industry. 

Reorganize  current  staff  functions  for  greater  use  of  experi¬ 
enced  personnel  in  reviewing  and  monitoring  the  preparation 
of  requirements,  work  statements,  program »plans,  nnd^ 
requests  for  proposals  during  the  Conceptual  and  Definition 
Phases.  f 

•  Adopt  the  proposed  Air  Force  regulation  of  Task  Group  1^ 
establishing  an  Air  Force-wide  System  Effectiveness  Program 
and  applying  the  implementation  steps  suggested  by  Task 
Group  V.  ^ 

•  Initiate  an  in-house  study  to  produce  a  "how-to"  manual  for 
developing  quantitative  system  requirements. 

•  Initiate  an  in-house  study  to  incorporate  the  content  of  the  various 
WSF.IAC  Task  Group  reports  into  a  scries  of  "how-to"  manuals. 

•  Adopt  the  mathematical  framework  of  WSEIAC  (Task  Groups  II 
and  IV)  and  the  activity  networks  of  this  report  and  incorporate 
them  into  the  AFSC  375  series  System  Management  Manuals 
and  into  techniques  manuals  which  relate' to  system  effective¬ 
ness. 

Develop  and  standardize  definitions  for  system  effectiveness 
and  related  terms  such  as  those  contained  in  the  Glossary  of 
Effect  ivenes s /Cost -Effect ivenes s  Terms,  Appendix  III  of  this 

report. 
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•  Adopt  the  charter  and  suggested  organization  proposed  by 
Task  Groups  III  and  V  initiating  the  establishment  of  a 
System  Effectiveness  Information  Central  (SEIC)  and  System 
Information  Banks  (SIB's)  for  Air  Force  systems. 

•  Define  methods  of  establishing  meaningful,  minimum 
acceptable  requirements  when  they  must  be  stated  as 

y  ■  1 

probabilities. 

•  Define  a  minimum  list  of  system  effectiveness/cost- 
effectiveness  data  elements  for  each  Air  Force  system  and 
take  steps  to  incorporate  these  elements  irfeufyeht  data 
collection  systems. 

•  Refine  the  use  of  system  models  in  the,  management  decision 
process  with  particular  attention  to  the  establishment  of 
management  guides  to  model  output  interpretation  (decision 
algorithms). 

•  Initiate  action  to  service  test  the  system  effectiveness /cost- 
effectiveness  prediction  techniques  proposed  by  the  WSEIAC 
on  one  or  more  current  and  proposed  Air  Force  systems. 

Finally,  Headquarters  AFSC  should  form  a  committee  to  review  each 
of  the  WSEIAC  task  group  reports  in  detail  and  to  formulate  long  range  plan 
for  implementation. 


SECTION  VII 


IMPACT  ON  EXISTING  DISCIPLINES 

,«P 

A.  A  NOTE  OF  CAUTION 

A  real  concern  of  many  people  is  the  relationship  of  System/Cost  - 
Effectiveness  to  the  other  disciplines  and  further,  its  relationship  to  the 
AFSC  375  series  Systems  Management  Manuals.  If  System/Cost  - 
Effectiveness  analysis  is  an  all-encompassing  discipline,  then  where  do 
the  other  specialties  such  as  reliability  and  maintainability  engineering  fit 
in?  Are  these  disciplines  eliminated?  Definitely  not  I  Consider  an 
analogous  situation.  We  intend  to  build  an  airplane;  then  we  need ‘aeronauti¬ 
cal  engineers.  Do  we  not  also  need  structural  specialists,  and  the  like? 
Clearly,  these  specialists  play  a  role  in  creating  the  airplane-as-a-system. 
When  .individual  system  effectiveness  disciplines  such  as  reliability,  human 
engineering,  safety  engineering,  value,  etc.  ,  must  still  be  amplified  by 
unique  specifications,  the  functional  flow  system  of  the  AFSC  375  series 
provides  a  framework  for  relating  these  individual  elements  to  each  other 
and  to  the  mainstream  of  merged  technological  and  management  effort  that 
flows  from  the  beginning  to  the  end  of  each  program.  » 

The  next  question  that  logically  arises  is,  "Will  we,  by  integrating 
these  disciplines  and  specializations,  be  adding  some  all-powerful  super¬ 
structure?  Will  we  have  to  cope  once  more  with  all  the  funding,  manning, 
communications  and  other  problems  associated  with  the  addition  of  a  new 
fragment  to  an  established  organization?"  The  answer  is  neither  a  clear 
yes  nor  a  definite  no.  In  some  instances,  the  function  already  exists  --  not 
as  System/Cost -Effectiveness  per  se,  but  as  some  combination  of  loosely 
related  elements  of  the  management  structure.  These  may  include  product 
assurance,  reliability,  systems  analysis,  etc.,  each  placing  some  emphasis 
on  cost-effectiveness.  If  the  function  is  there  in  one  of  these  impalpable 
forms,  or  if  the  function  is  missing  entirely,  system  effectiveness  prob¬ 
lems  which  arise  are  not  being  cured  except  in  an  irrational,  unscientific, 
ineffective  manner  which  costs  too  much  in  time,  performance  and  money. 
Thus,  organization  for  system  effectiveness  is  necessary.  Funding, 
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manning  and  training  for  a  "new"  function  called  system/cost-effectiveness 
will,  probably,  mean  change  and  reorientation.  But  if  this  means  increased 
performance,  reduced  costs,  reduced  development  time  and  more  oderli- 
ness  to  a  program,  then  such  a  change  should  be  welcomed. 

B.  LIMITATIONS 

Cost-effectiveness  analysis  must  be  applied  carefully.  It  is  not  a 
panacea  nor  is  it  a  new  technique.  It  has  been  a  part  of  military  planning 
for  years.  But  the  complexity  of  military  tasks  now  requires  a  multi¬ 
disciplinary  and  more  systematic  approach.  The  major  utility  of  cost- 
effectiveness  analysis  is  that  it  uses  all  the  available  knowledge  and  data  in 
as  efficient  and  complete  a  manner  as  is  possible  to  give  management  ' 
information  needed  for  decision  making. 

If  they  are  to  be  of  value,  the  results  of  cost-effectiveness  studies 
must  be  given  in  terms  meaningful  to  those  who  make  decisions  and  under¬ 
stand  the  implications  and  results  of  these  analyses.  Thus,  the  analyst 
should  appreciate  the  problems  of  communication  with  a  broad  spectrum  of 
people  including  design  engineers,  company  managers,  military  managers, 

military  planners  and,  sometimes,  congressmen  and  the  general  public. 

* 

In  interpreting  the  results  of  the  studies,  it  must  be  remembered  that 
the  state-of-the-art,  resource  constraints,  political  and  military  thinking 
and  philosophy,  enemy  posture,  etc.,  are  in  a  constant  state  of  flux.  Thus, 
these  results  should  not  become  associated  with  hard,  fast,  unchanging  rules. 
A  current  finding  that  a  reliability  of  0.9  is  best  for  a  particular  component 
should  not  become  permanent  dogma.  The  results  should  never  be  the  basis 
for  hindering  research.  Rather,  they  should  provide  guidelines  for  further 
exploration  or  tests  designed  to  yield  more  fruitful  information  on  which  to 
base  decisions. 

The  limitations  of  cost-effectiveness  studies  have  already  been  sug¬ 
gested  in  the  foregoing  paragraphs.  The  reader  should  bear  in  mind  that, 
whatever  shortcomings  or  dangers  may  be  associated  with  analytical  studies 
such  as  those  proposed  by  the  WSEIAC,  decisions  based  on  intuition,  experi¬ 
ence  which  has  not  been  thoroughly  analyzed,  or  a  sample  of  personal  opinion 
(deep-seated  feeling)  are  certainly  less  defensible  and  more  subject  to 
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omissions  of  important  factors.  One  would  not  build  a  bridge  by  intuitive 
design,  overlooking  sound  structural  engineering  practice;  yet,  many 
unknowns  exist  in  regard  to  material  mechanics  and  random  loading  be- 

k  ; 

havior  of  structures. 

Although,  in  a  sense,  statements  on  limitations  of  cost-effectiveness 
analyser  may  be  regarded  as  platitudes,  we  present  some  of  them  hei-'e  aS' 
reminders. 

•  Cost-effectiveness  indices  cannot  be  meaningful  unless  derived 
from  a  model  which  represents  the  "real  world"  fairly  closely.  Reality 
should  not  be  buried  under  mountains  of  detail  nor  does  great  detail,  by 
itself,  create  reality  in  a  model. 

•  It  must  be  remembered  that  cost-effectiveness  analysis  is  an 
iterative  process.  Early  results  should  not  be  permitted  to  create  such  a 
lasting  impression  (favorable  or  unfavorable)  as  to  lead  one  to  ignore  the 
results  of  later  refinements.  This  could  lead  to  disillusionment  on  the  part 
of  all  concerned  and  eventually  to  abandonment  of  a  valuable  tool. 

•  Cost-effectiveness  analysis  can  never  replace  good  engineering 
and  management  practices.  It  should  be  regarded  as  a  supplementary  tool 
to  provide  meaningful  information.  Final  decisions  must  still  be  based  upon 
sound  judgment.  This  must  be  particularly  emphasized  since  too  many  poli¬ 
tical,  psychological  (e.g.  ,  an  individual's  drive  to  solve  a  particular  prob¬ 
lem),  prestige  value,  and  other  factors  cannot  be  considered  in  a  satisfac¬ 
tory  manner  at  this  time  in  such  analyses. 

"  When  results  are  sensitive  to  factors  associated  with  high  degrees 
of  risk  or  uncertainty,  "warning  signs"  must  be  posted.  The  results  must 
then  be  used  judiciously  in  making  decisions. 

In  much  of  what  has  been  said  earlier,  there  is  an  obvious  attempt  to 
build  up  the  importance  of  cost-effectiveness  consciousness.  Considerable 
emphasis  has  been  placed  on  developing  models  for  obtaining  cost- 
cffcctivcncss  indices  and  optimization  thereof.  However,  it  must  be  remem¬ 
bered  that  these  do  not  provide  a  final  answer.  They  do  provide  guidelines, 
but  judgment  must  still  play  a  large  part. 
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Perhaps  this  is  best  expressed  by  Dr.  Alain  Enthoven's  statement:-^ 
"Do  judgment  and  experience  have  no  place  in  this  approach  to  choice  of 
weapon  systems  and  strategy  and  design  of  the  defense  programs?  Quite 
the  contrary.  The  statement  that  the  issue  is  judgment  versus  computers 
is  a  red  herring.  Ultimately  all  policies  are  made  and  all  weapon  systems 
are  chosen  on  the  basis  of  judgments.  There  is  no  other  way  and  there 
never  will  be.  The  question  is  whether  those  judgments  have  to  be  made  in 
the  fog  of  inadequate  and  inaccurate  data,  unclear  and  indefinite  issues,  and 
a  welter  of  conflicting  personal  opinions,  or  whether  they  can  be  made  on 
the  basis  of  adequate,  reliable  information,  relevant  experience,  and  clearly 
drawn  issues.  The  point  is  to  render  unto  computers  the  things  that  are 
computers'  and  to  judgment  the  things  that  are  judgment's.  In  the  end,  there 
is  no  question  that  analysis  is  but  an  aid  to  judgment  and  that,  as  in  the  case 
of  God  and  Caesar,  judgment  is  supreme." 

Thus,  although  there  are  limitations  in  this  modeling  process  to  obtain 
cost-effectiveness  indices,  it  must  be  remembered  that  this  approach  allows 
us  to: 

•  organize  and  set  into  proper  perspective  the  many  alternatives 

of  the  problem;  * 

•  establish  many  "if-then"  statements,  pertaining  to  the  alter¬ 
natives  of  the  problem; 

•  evaluate  properly  data  uncertainties; 

•  examine  many  cases  quickly  which  would  require  years  of 
simulated  combat  to  test;  and 

•  explore  systematically  those  cases  which  cannot  be  tested 
(you  cannot  go  to  war  to  test  system  effectiveness). 

And  another  caution.  There  are  still  many  unsolved  problems.  The 
task  group  reports  arc  not  "how-to"  manuals.  However,  WSEIAC  has 

— ^  From  a  lecture,  "Decision  Theory  and  Systems  Analysis,"  delivered 

during  the  Distinguished  Lecture  Series,  sponsored  by  the  Board  of  Trade 
Science  Bureau,  Washingon,  D.  C.,  December  5,  1963. 


provided  more  than  the  nucleus  of  a  solution. 

Many  positive  recommendations  have  been  presented.  These  include 
a  summarization  of  current  cost-effectiveness  techniques  and  a  task  analysis 
that  identifies  critical  activities  to  be  meshed  with  the  current  Air  Force 
management  series  of  manuals  as  an  activity  network  governing  system 
effectiveness  assurance. 
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TASK  ANALYSIS  OF  A 

SYSTEM  EFFECTIVENESS/COST-EFFECTIVENESS 
PREDICTION/EVALUATION/ AUGMENTATION  CYCLE 

A.  INTRODUCTION 

It  is  the  purpose  of  this  appendix  to  amplify  the  earlier  discussion  of 
the  fifteen  steps  for  conducting  a  System  Effectiveness/Cost-Effectiveness 
Prediction,  Evaluation  and  Augmentation  Cycle.  Each  step  corresponds  to 
a  block  in  Figure  1.  Since  there  is  a  considerable  amount  of  detail,  a  more 
detailed  foldout  of  Figure  1  has  been  inserted  (Figure  14)  to  give  the  total 
task  sequence  on  one  chart.  It  will  be  helpful  to  fold  out  Figure  14,  page  125, 
at  this  time. 

Each  step  is  discussed  in  detail  and  in  numerical  succession, 
indicating  the  general  content  of  each  block,  problem  areas  which  exist, 
and,  whenever  possible,  recommended  solutions.  Pertinent  considerations 
appropriate  to  the  various  phases  of  system  life -cycle  are  also  included. 
Following  this  exposition,  an  example  analysis  applicable  to  Conceptual 
Phase  studies  is  given  in  Appendix  II  to  illustrate  the  task  analysis. 


B.  BLOCK  1.0  MISSION  DEFINITION 

The  output  of  the  mission  selection  analyses  conducted  in  the  Concep¬ 
tual  Phase  and  pre -Conceptual  Phase  is  a  statement  of  what  the  system  is  to 
accomplish.  Historically,  this  output  covers  the  end-item  functions  which 
are  to  be  accomplished  {target  destruction,  reconnaissance,  9pace  explora¬ 
tion,  etc.)  and  the  conditions  and  geographic  locations  within  which  these 
functions  are  to  take  place.  The  optimization  processes  that  take  place  in 
the  succeeding  Definition  and  Acquisition  Phases  consist  mainly  of  synthe¬ 
sizing  alternate  means  of  meeting  stated  objectives,  evaluating  them,  and 
selecting  the  combination  of  such  alternatives  which  secures  the  most 
favorable  cost-effectiveness  relationship. 

Accordingly,  it  follows  that  if  the  statement  of  program  objectives 
severely  limits  the  alternatives  which  can  be  considered,  the  ability  to 
achieve  optimum  cost-effectiveness  during  later  phases  is  correspondingly 
reduced.  Thus,  the  statement  of  program  objectives  and  mission  derived 
from  studies  during  the  Conceptual  Phase  analyses  should  tend  to  define  what 
is  to  be  done  rather  than  specifically  direct  how  the  task  is  to  be  accom¬ 
plished.  It  should  be  noted,  however,  that  additional  constraint  and 
relationship  data  must  be  provided  in  order  that  systems  and  resource  use 
selection  correspond  to  the  basis  on  which  the  mission  was  initially  justified. 

It  is  a  fundamental  requirement  of  the  methods  recommended  by  the 
WSEIAC  that  a  clear  and  unambiguous  statement  of  the  mission  of  a  system 
be  obtained.  This  definition  should  contain: 

1.  1  functional  description  (purpose)  of  system,  and 

1.2  system  quantitative  requirements. 

A  functional  description  of  the  system  should  be  directly  derivable 
from  an  ROC  or  QOR.  Although  this  description  may  be  somewhat  modified 
at  the  end  of  the  Acquisition  Phase,  the  ROC  and  QOR  should  remain  the 
fundamental  reference. 

The  WSEIAC  has  identified  a  problem  area  with  respect  to  the  estab¬ 
lishment  of  quantitative  requirements.  Some  system  requirements  (e.g., 
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riTigc)  will  tely  frnrr.  t-Vi  <a  ROr  nr*  QflR.  'For  th«BB,  the 

statement  of  minimum  acceptable  performance  levels  will  be  straight¬ 
forward,  For  others,  especially  those  for  composite  system  attributes 
such  as  reliability  and  availability,  minimum  acceptable  values  will  not  be 
evident  from  the  ROC  or  QOR.  In  these  latter  cases  the  WSEIAC  recom¬ 
mends  that  comparative  analysis  be  employed  during  the  Conceptual  Phase 
to  establish  tentative  requirements.  Such  requirements,  documented  in  a 
TSOR  become  the  guiding  input  to  the  Definition  Phase  where  they  can  be 
further  refined  to  become  a  firm  set  of  requirements  in  the  SOR, 

The  problem  area  that  the  WSEIAC  has  identified  in  this  process  is: 

•  There  is  no  generally  accepted  set  of  techniques  for 
determining  minimum  acceptable  requirements  that 
holds  for  all  the  factors  of  effectiveness,  particularly 
for  those  factors  which  are  stated  as  probabilities. 

This  problem  may  be  paraphrased  as:  "How  much  is 
enough? " 


c. 


BLOCK  2.0  RESOURCES 


Resources  usually  evidence  themselves  as  a  practical  constraint  on 
the  development  and  procurement  of  a  system.  There  are  four  principal 
areas  of  consideration  here: 

2.  1  budget 

2.  2  SPO  manning 

2.  1  industry  capacity  \ 

2.4  technology. 

Current  budgetary  practices  constitute  a  limitation  on  the  full 
exploitation  of  the  System  Effectiveness /Cost  - Effeetivenes s  prediction 
techniques  currently  available.  Specifically: 

'  A  trade-off  based  on  System  Effectivcness/Cost-Effectiveness 
prediction  techniques  in  the  Conceptual  Phase  callB  for  a 
balance  between  investment  costs  (AFSC)  and  support  costs 
(AFLC  and  using  command)  for  the  life  of  the  system.  This 
is  contrary  to  current  budgetary  practices  which  allocate 
funds  by  command  and  by  program  element. 

•  Trade-off  studies  based  on  System  Effectiveness/Cost  - 
Effectivcness  prediction  techniques  in  the  Definition  and 
Acquisition  Phases  require  recognition  that  greater 
expenditures  from  the  AFSC  pocket  can  result  in  less 
expenditures  for  the  using  and  support  commands.  Past 
budgetary  practices  did  not  encourage  consideration  of 
this  attitude. 
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D.  BLOCK  3.0  SYSTEM  DESCRIPTION 

The  depth  of  system  description  {and,  consequently,  the  detail  in 
System  Effectiveness/Cost-Effectivenes s  prediction)  depends  upon  the  phase 
of  the  system  life  cycle.  System  description  consists  of  either: 

■*.  1  identification  of  alternative  system  configurations,  or 

i 

3.2  configuration  documentation,  followed  by 

3.  3  system  summary  description. 

The  ability  to  optimize  a  system  depends  on  the  availability  of  alter¬ 
nate  means  of  meeting  the  requirements.  Alternatives  include  the  means, 
approaches,  or  techniques  which  can  be  employed  to  meet  the  stated  require¬ 
ments  within  the  constraints  of  the  resources.  Obviously,  if  no  alternatives 
present  themselves,  or  if  they  are  ruled  out  by  the  statement  of  require- 
ments  and  resources,  there  is  no  problem  in  selection.  It  also  follows  that 
when  alternatives  do  present  themselves,  decision  between  them  is  required. 
If  the  system  is  to  be  optimized  with  respect  to  cost-effectiveness,  then  the 
optimization  process  must  extend  to  each  decision  made  on  the  alternatives 
presented.  • 

Table  I  shows  an  example  of  some  of  the  types  of  alternatives  con¬ 
sidered  in  optimization  studies. 

It  is  possible  to  arrive  at  the  optimum  system  of  a  given  type  by  de¬ 
signing  a  great  number  of  alternative  systems,  estimating  cost  and 
effectiveness  for  each,  and  simply  selecting  the  best  one.  However,  the 
large  number  of  man-hours  required  to  do  this  renders  such  an  approach 
impractical.  A  more  practical  approach  is  to  consider  only  a  very  few  basic 
configurations  or  candidate  systems  within  a  given  system  type.  A  com¬ 
pletely  adequate  cost-effectiveness  optimization  of  the  system  can  often  be 
ac  complished  with  as  little  as  one  basic  configuration.  However,  due  to 
the  small  number  of  basic  configurations  thus  explored,  it  is  necessary 
that  each  basic  configuration  be  optimized  within  itself.  This  is  accomp¬ 
lished  by  synthesizing  and  evaluating  variations  or  alternatives  at  several 
levels  within  the  basic  configuration.  These  alternatives  may  take  the  form 
of  cither  physical  or  performance  characteristics. 


TABLE  I 


TYPICAL  ALTERNATIVES 
POSSIBLE  IN  COST -EFFECTIVENESS 
OPTIMIZATION 

Basic  Concept 

Manned  Versus  Unmanned 
Liquid  Versus  Solid  Rockets 
System  and  Subsystem  Type 

Battery  power  versus  generation 
Materials  Choice 

System  and  Subsystem  Configuration 
Redundancy 
Maintenance 

Hi -Reliability  versus  MIL  Std  Parts 
Operational  modes 
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Each  military  system  has  a  number  of  physical  characteristics  that 
affect  cost,  performance  and  effectiveness.  A  list  of  physical  characteris¬ 
tics  to  cover  all  systems  will  not  be  attempted  here.  A  few  of  those  common 
to  most  systems  include  weight,  volume,  shape,  energy  levels,  mechanical 
and  electrical  packaging,  and  environmental  capabilities.  The  physical 
characteristics  of  a  system  affect  the  cost  elements  incurred  in  development, 
procurement  and  support.  There  is  obviously  a  broad  range  of  cost  sensi¬ 
tivity  as  cost  elements  are  compared  for  different  design  alternatives  of  a 
given  system  requirement  as  well  as  for  different  technology  alternatives 
within  a  given  design  alternative.  * 

When  one  considers  the  area  of  performance  characteristics  of  military 
systems,  it  is  difficult  to  prepare  a  comprehensive  listing,  and  few  perfor¬ 
mance  characteristics  are  common.  Typical  performance  characteristics 
for  a  few  military  systems  include:  accuracy,  speed,  thrust,  memory 
capacity,  computational  capability,  signal  to  noise  ratio,  range,  power  out¬ 
put,  discrimination,  etc.  Relationships  between  cost  elements  and  perfor¬ 
mance  characteristics  are  fertile  areas  for  optimization.  A  particular  cost 

© 

element  will  vary  as  the  performance  characteristic  varies  over  the  range 
of  values  possible  for  the  design  alternative.  For  a  given  requirement  level 
of  a  performance  characteristic,  cost  variation  as  a  function  of  the  different 
design  and  technology  alternatives  within  a  design  alternative,  are  of  prime 
importance.  The  constraints  on  performance  characteristics  are  generally 
set  by  scientific,  engineering,  and  manufacturing  knowledge  and  capabilities. 

In  listing  the  alternatives,  primary  importance  should  be  given  to  those 
which  have  a  significant  impact  on  cost  or  the  resources  established  in  the 
statement  of  requirements.  A  preliminary  analysis  of  an  initial  system 
design  can  ordinarily  indicate  the  major  impact  areas. 

The  number  of  alternatives  to  be  considered  in  the  optimization  process 
can,  in  many  cases,  be  reduced  by  screening  these  alternatives  against  the 
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available  resources  established  in  the  statement  of  requirements.  In  the 
area  of  cost,  physical  characteristic  constraint  relations  established  outside 
the  cost  area  will  often  bound  and  limit  the  feasibility  nr  scope  of  alternatives. 

As  an  example  of  such  screening,  let  us  look  at  a  case  wherein  an  isotope 
power  source  is  being  considered  as  an  alternative  to  a  power  system  design 
more  compatible  with  current  state-of-the-art.  If  the  required  date  for 
system  operational  capability  is  relatively  early  in  time,  the  isotope  power 
source  may  be  automatically  ruled  out  by  lack  of  availability  by  the  required 
date . 

An  example  of  an  alternate  type  of  screening  problem  could  occur  when 
comparing  the  same  isotope  energy  source  against  an  operational  date  stated 
as  a  variable.  Assuming  that  system  effectiveness  or  value  decreases  as 
the  operational  date  is  delayed,  it  may  be  possible  to  eliminate  the  isotope 
energy  source  from  further  detailed  consideration  on  the  basis  that  the  cost 
or  effectiveness  gains  associated  therewith  do  not  compare  favorably  with 
the  value  or  effectiveness  lost  due  to  the  corresponding  slide  in  operational 
date. 

In  preparing  a  list  of  alternatives,  one  should  associate  them  with  the 
level  at  which  decisions  upon  the  alternatives  are  to  be  made.  At  system 
level,  decisions  should  be  made  on  alternatives  which  impact  on  the  basic 
system  configuration  or  operational  mode.  Decisions  which  do  not  directly 
or  substantially  affect  basic  configuration  and  operational  mode  should  be 
made  at  lower  levels  using  trade-off  factors  developed  for  the  entire  system. 
If  such  lower  level  decisions  are  attempted  as  a  part  of  the  over-all  system 
optimization  process,  the  scope  of  the  system  level  problem  may  become 
unmanageable.  It  is  recognized,  however,  that  the  basic  system  may  change 
significantly  as  a  result  of  optimizations  at  subsystem  level.  Further,  the 
trade-offs  and  optimizations  made  at  subcontractors  level  with  a  single  sub- 
assembly  or  black  box  may  have  far-reaching  effect  on  system  effectiveness. 
Thus,  any  system  for  handling  cost-effectiveness  must  permit  optimization 
to  feed  both  up  and  down  through  the  various  system  levels  and/or  tiers  of 
customer /contractor /subcontractor .  The  process  of  feeding  up  and  down 
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through  the  system  mu9t  be  recognized  as  an  iterative  one,  wherein  it  may 
be  necessary  to  reiterate  some  of  the  lower  level  suboptimizations  to  insure 
that  the  basic  system  changes  have  not  altered  previously  established  con¬ 
clusions. 

During  the  Conceptual  Phase,  steps  3.  1  (identify  alternative  system 
configurations)  and  3.  3  (system  summary  description)  form  a  logical 
sequence  for  system  description.  In  the  late  Definition  Phase  and  Acquisi¬ 
tion  Phase,  the  emphasis  increasingly  shifts  to  3.2  (configuration 
documentation)  followed  by  3,  3.  The  latter  activity,  common  to  both 
.sequences,  contains  as  a  minimum: 

1.3.  1  specification  of  levels  of  system  organization; 

2  / 

3.  3.2  spocifi.cnt.ion  of  STOC--  and  their  time  lines  by 
equipment /function; 

3,3.3  specification  of  physical  factors; 

3.  3.4  specification  of  support  policy  by  type  and  time  line;  and 

3.3.5  specification  of  u«e  plan. 

This  summary  description  must  reflect  all  those  features  of  system 
structure  which  can  affect  either: 

•  the  estimation  of  effectiveness,  or 

•  a  system  effectiveness/cost-effectiveness  trade-off 

analyBie. 

The  distinction  here  between  effectiveness  and  cost-effectiveness  is 
important.  A  particular  system  feature  may  be  uncontrollable,  and  hence 
not  capable  of  manipulation  for  coot-effrctiveness  trade-off  analysis,  but  it 
must  bo  included  in  an  effectiveness  calculati  n  in  order  to  estimate  current 
or  predicted  status. 


2/ 


Standard  Tactical  Operating  Conditions 
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During  the  Definition  and  Acquisition  Phases  the  system  is  increasingly 
defined  on  paper.  On  an  iterative  basis,  the  following  activities  of  configu¬ 
ration  documentation  (3.2)  occur; 

3.2.1  perform /review  function  analysis; 

3.2.2  make /review  engineering  drawings; 

3.2.  3  assemble /review  physical  factors  summary  document;  they 

are  conducted  on  an  "as  pertinent"  basis. 

These  activities,  taken  jointly,  lead  to  further  detailed  documentation 

activities: 

3.  2.  4  perform/review  equipment  operating  time  line  analysis  ; 

3.  2.  5  assemble/review  integrated  task  index  ; 

3.2.6  assemble /review  unit  manning  document; 

3*2.7  assemble/review  reliability  indices  report; 

3.2.8  assemble/review  data  handbook; 

3.2.9  assembte/review  provisioning  requirements  document; 

3.2.  10  assemble /review  cost  indices  document; 

3.  2.  11  assemble/review  planning  factors  document; 

The  content  of  these  activities  must  be  reflected  in  (3.  3)  system  summary 
description. 

The  WSEIAC  has  emphasized  by  illustrative  example  that  each  of  the 
above  activities  is  essential  to  an  adequate  system  description.  A  problem 
area  has  been  identified  in  the  area  of  cost  indices  documentati  -a.  Currently, 
cost  data  is  not  summarized  in  a  form  suitable  for  use  in  Syst<  n  Effective¬ 
ness /Cost -Effectiveness  calculations. 
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E. 


BLOCK  4.0  FIGURES  OF  MERIT 


A  figure  of  merit  is  a  statement  which  relates  mission  objectives 
(ROC  or  QOR)  to  quantitative  system  requirements  (SOR).  It  is  a  statement 
of  the  ability  of  a  system  to  meet  an  operational  need,  including  the  recog¬ 
nition  of  the  risk  and  uncertainty  that  are  fundamental  characteristics  of  the 
military  mission. 

Risk  is  synonymous  with  odds.  Recognition  of  risk  leads  to  the  state¬ 
ment  of  a  figure  of  merit  in  probabilistic  terms. 

Uncertainty  is  associated  with  lack  of  knowledge  or  empirical  data 
necessary  to  establish  value  or  range.  Recognition  of  uncertainty  leads  to 
the  consideration  of  the  possible  (unpleasant)  surprises  of  the  future.  Such 
considerations  reflect  themselves  in  alternative  (potential)  mission  objec¬ 
tives  and  alternative  figures  of  merit. 

1 .  Principal  Measure 

The  most  comprehensive  figures  of  merit  are  system  effective¬ 
ness  and  cost-effectiveness.  It  is  the  purpose  of  this  section  to  describe  the 
WSEIAC  concepts  of  effectiveness  and  cost-effectiveness  in  some  detail. 

System  effectiveness  is  a  quantitative  measure  of  the  extent  to 
which  a  system  may  be  expected  to  achieve  a  set  of  specific  mission  require¬ 
ments. 

System  effectiveness  prediction/evaluation  is  based  upon 
probabilistic  concepts.  A  suitable  format  for  expressing  system  effective¬ 
ness  is  "the  probability  (x)  that  a  system  can  successfully  accomplish  a 
specific  mission  directive  at  a  random  point  in  time,  following  the  establish¬ 
ment  of  an  alarm  condition,  shall  be  greater  than  (y)  with  probability  (z)." 
This  statement  is  meaningful  only  in  relation  to  a  test  program  and  hence 
implies  that  the  SOR  requires  a  minimum  acceptable  demonstration  test 
program. 

System  effectiveness  may  be  regarded  to  be  a  function  of  three 
major  system  attributes:  availability  (A),  dependability  (D)  and  capability  (C). 

Availability  (A)  is  a  measure  of  the  system  condition  at  the  start  of 
the  mission,  when  the  mission  is  called  for  at  an  unknown  (random)  point  in  time. 
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This  factor  of  effectiveness  has  also  been  referred  to  in  the  past  (not 
always  accurately)  as  Operational  Availability,  Operational  Readiness, 
Alert  Readiness,  Ready  Rate,  and.  Real  In-Commission  Rate.  It  is  usually 
expressed  as:  (See  Glossary)  * 


t. 

1 


where 

T.  =  mean  time  between  system  interruptions 

Ta  =  mean  time  assigned  to  the  up  condition 

=  mean  time  assigned  to  the  down  condition. 

A  system  interruption  is  defined  to  be  anv  event  which  removes  the  system 
from  its  alert  status.  Thus,  failures  (known  or  unknown),  planned  or  un¬ 
planned  maintenance,  and  administrative  actions  (such  as  crew  training 
exercises)  are  all  potential  causes  of  system  interruption. 


* 

An  estimate  (A)  of  steady  state  availability  can  be  obtained  by 
calculating: 


total  time  system  is  in  true  up  condition 
total  calendar  time  of  observation  of  system 


The  denominator  of  this  expression  is  directly  observable.  The  numerator 
however,  is  directly  observable  only  when  all  failures  are  observable  at  (or 
near)  the  instant  of  occurrence.  In  general,  this  will  not  be  the  situation, 
and  in  some  cases,  fairly  complicated  methods  of  statistical  inference  may 
have  to  be  employed  to  estimate  the  true  status  of  the  system. ^ 

Availability  is  calculated  as  the  expected  fraction  of  time  that  a  system 
is  in  an  operable  condition  in  a  specified  time  interval. 

Availability  means  the  probability  that  a  system  will  be  operable  and 
ready  to  initiate  a  mission  at  a  random  point  in  time. 
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Availability  encompasses: 

•  system  failure  rates  , 

.  system  repair  rates, 

•  system  maintenance  policy /procedures, 
personnel  factors, 

•  system  support  policy. 

i  ; 

Proper  expressions  for  availability  always  reflect  explicit  measures 
of; 

personnel, 

•  procedures, 

•  hardware.  . 


Dependability  is  a  measure  of  the  system  condition(s)  at  one  or  more 

points  during  the  mission;  given  the  system  condition(s)  at  the  start  of  the 

mission.  It  will  usually  be  stated  as  the  probability  (or  probabilities  or 

other  suitable  mission  oriented  measure)  that  the  system  will  enter  and/or 

occupy  any  one  of  its  significant  states  during- a  specified  mission.  It 

specifically  includes,  but  is  not  limited  to,  equipment  reliability  during  the 

mission.  For  example,  if  repair  is  possible  during  a  mission,  dependability 

will  include  the  effects  of  the  repair  capability  on  mission  success.  On  the 

other  hand,  if  no  repair  potentiality  exists,  but  there  are  backup  modes  of 

performance,  proper  expressions  for  dependability  must  specifically  account 

for  the  probabilities  of  requiring  the  use  of  the  alternative  (and  possibly 

(3  71 

degraded)  modes  of  performance.'  ’  ' 


The  formulation  of  dependability  expressions  is  strongly  influenced  by 
the  type  of  system  and  system  utilization;  thus,  no  single  generalized  ex¬ 
pression  can  be  written  for  dependability. 

Dependability  accounts  for  a  variety  of  system  aspects,  for  example: 


reliability, 


ground  vulnerability 
flight  vulnerability 


l 

J 


penetrability , 


r  epair  ability . 


(survivability) , 
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The  meaning  of  dependability  will  vary  depending  upon  the  significance 
attached  to  the  various  system  conditions  which  can  occur  during  the  mission. 
For  example,  the  dependability  expressions  for  an  ICBM  are  simply  the 
probabilities  of  successful  survival,  launch,  flight,  and  penetration  under 
tactical  conditions.  For  a  manned  aircraft  with  multimode  delivery  capabi¬ 
lity,  the  dependability  expressions  will  usually  be  the  probabilities  of  being 
required  to  use  each  of  the  various  possible  modes  of  weapon  delivery.  On 
the  other  hand,  the  time  spent  in  each  possible  system  state  during  the 
mission  may  be  the  crucial  factor  in  dependability.  In  this  case,  the  depend¬ 
ability  expressions  may  be  chosen  to  be  either  the  probability  of  spending  a 
time  "  t  "  in  any  given  state,  or  the  expected  fraction  of  time  spent  in  any 
given  state  during  the  mission.  .  ' . 

Survivability  is  the  probability  that  a  system  will  either  (1)  be 
removed  from  the  threatened  environment  before  it  can  be  attacked  (as  with 
warning),  or  (2)  "ride  out'-  some  anticipated  attack.  In  the  first  instance, 
the  basic  parameters  are  the  amount  of  reliable  warning  time  and  the  reac¬ 
tion  time  from  "alerted"  through  reaching  a  "safe"  environment.  In  the 
latter  case,  which  is  commonly  assumed  for  hard-site  ballistic  missiles, 
many  things  can  become  important:  blast  hardness  (i.e.  ,  resistance„to 
overpressure);  electronic  hardness;  dispersal  mobility;  deception;  active 
defense;  and  above  all,  the  weight  or  severity  of  the  expected  attack.  If  a 
weapon  system  is  intended  to  provide  a  credible  deterrent  threat  for  a  sub¬ 
stantial  time  (say  days  or  even  weeks)  after  initiation  of  hostilities,  it  must 
not  only  survive  possible  missile  attacks,  but  perhaps  also  manned -bomber 
attacks.  In  addition,  such  extensive  periods  of  operation  must  be  supported 
in  the  likely  absence  of  "normal"  (i.e.,  peacetime)  services  like  commer¬ 
cial  power,  telephones,  and  even  highway  travel. 

Reliability  is  the  probability  that  an  "available"  system  (i.  e.  ,  'in- 
commission  and  having  no  hidden  defects  detectable  by  monitoring  or 
periodic  checkout)  will  operate  without  failure  during  the  mission.  In  the 
case  of  an  I.CBM,  the  reliability  aspect  of  dependability  is  usually  considered 
to  be  the  product  of  launch  reliability  and  flight  reliability.  Launch  reliability 
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can  be  thought  of  as  being  made  up  of  Command  Reliability  (accomplish  the 

specified  pre -launch  procedures  within  the  allotted  time),  and  Initiation 
Reliability  (perform  the  irreversible  or  “one-shot"  sequence  of  launch  events 
such  as  firing  squibs,  door  ordnance,  igniters,  etc.).  All  three  of  these  sub- 
elements  depend  also  in  some  fashion  on  the  quality  of  maintenance  activities 
accomplished  during  the  period  of  strategic  alert.  In  other  words,  they  are 
affected  by  the  same  things  as  the  availability  expression  though  not 
necessarily  in  the  identical  manner.  For  example,  the  pre-launch  proce¬ 
dures  are  often  similar  and  sometimes  identical  to  periodic  exercises 
performed  for  verification  of  alert  status. 

For  ballistic  missiles,  flight  reliability  ordinarily  includes,  in  addition 
to  propulsion  and  control  and  other  factors,  proper  engine  cut-off,  staging, 
and  guidance.  For  manned  aircraft,  both  hardware  performance  and  human 
performance  (correct  navigation  to  target,  aiming,  and  weapon  delivery)  are 
involved.  For  a  Jammer  System,  it  would  include  detection  of  enemy 
radiation,  selection  of  response  mode,  and  subsequent  radiation  of  the 
proper  jamming  signals.  Once  again,  the  hardware  reliability  is  related  to 
the  quality  of  maintenance  in  the  ground  environment. 

Penetrability  is  the  probability  that  a  weapon  system  will  survive  a 
defense  environment  and  arrive  at  the  target  intact.  For  manned  aircraft, 
this  probability  is  a  function  of  such  things  as  the  penetration  mode  (for 
example,  low  level  flight  to  avoid  detection),  speed,  maneuvers,  electronic 
countermeasures,  decoys,  etc.  For  ballistic  missiles,  for  example,  pene¬ 
trability  may  be  expected  to  be  100  per  cent  against  a  no-defense  environ¬ 
ment,  while  anti-ICBM  environments  make  penetration  aids  and  terminal 
maneuvers  important.  This  is  an  area  where  time  is  also  certain  to  be  a 
factor;  chronological  improvements  can  be  expected  in  the  quality  of  both 
offensive  and  defensive  tactics,  so  particular  levels  of  either  must  be 
associated  with  a  particular  point  in  time. 
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Capability  is  a  measure  of  the  ability  of  a  system  to  achieve  the  mission 

objectives;  given  the  system  condition(s)  during  the  mission.  It  specifically 

accounts  for  the  performance  spectrum  of  a  system.  For  example,  such 

>  3/ 

familiar  things  as  accuracy,  range,  payload,  lethality—  and  information 

(3) 

retrieval  rate  determine  the  capability  of  a  system. 

Like  dependability,  capability  is  clearly  peculiar  to  each  system  (and 
proposed  mission),  so  that  no  unique  set  of  expressions  applies  generally  to 
all  systems. 

In  general,  capability  expressions  are  a  measure  of  the  "worth"  or 
"value"  of  any  system  state.  The  measure  of  worth  can  be  stated  variously 
as  the  probability  of  accomplishing  the  mission  objectives  while  ip.  some 
given  state;  as  the  expected  number  of  targets  destroyed  per  sortie;  as  the 
probability  of  track,  given  detection;  and  so  forth. 

Of  the  three  factors  --  availability,  dependability,  and  capability  --  the 
last  is  usually  regarded  to  be  the  most  direct  expression  of  the  intent  of  an 
SOR.  If  availability  and  dependability  express  the  chances  of  getting  on  the 
ballot  --  capability  is  an  expression  of  the  ballot  count. 

2.  Additional  Considerations 

© 

In  addition  to  the  basic  factors,  availability,  dependability,  and 
capability,  there  are  certain  other  qualities  of  a  weapon  system  which  have 
an  impact  on  total  effectiveness,  though  they  are  less  susceptible  to  direct 
quantification.  For  example,  the  ability  to  retarget  a  ballistic  missile 


3/ 

—  Lethality  is  defined  here  as  the  probability  that  weapon  effects  will 
destroy  the  target.  For  a  point  target,  this  is  a  function  of  the  accuracy  of 
delivery,  usually  expressed  as  a  Circular  Error  Probability  (CEP),  and  the 
lethal  radius  (LR),  which  is  in  turn  a  function  of  warhead  yield,  burst 
altitude,  and  target  hardness.  For  area  targets,  lethality  can  be  related  to 
these  same  parameters  through  simple  nomograms,  which  can  give  (in 
addition  to  simple  probability  estimates)  the  expected  fraction  of  an  area 
target  that  will  be  destroyed.  (This  type  of  information  is  also  available 
for  multiple  warheads,  or  multiple  weapon  launches.)  The  latter  quantity 
can  be  considered  a  figure  of  merit  attributable  to  such  elements  as  yield, 
aim  point,  CEP  and  height  of  burstj(8) 
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quickly  and  simply  may  allow  a  reduction  in  the  extent  of  overlapping 
coverage  for  high-priority  targets,  and  thus  permit  cither  an  improvement 
in  the  long-term  coverage  of  secondary  targets,  or  alternatively,  a  reduc¬ 
tion  in  the  required  size  (and  cost)  of  the  total  force  structure.  The  inherent 
flexibilities  of  manned  systems  arc  likewise  significant. 

For  many  weapon  systems  safety  is  a  paramount  consideration. 

Unless  safety  features  arc  carefully  considered  during  the  development  pro¬ 
cess,  there  is  a  significant  probability  that  a  system  may  be  activated  by 
error  (operator,  maintenance,  spurious  signals,  failure  of  a  critical  circuit 
or  function,  etc.).  Military  and  strategic  consequences  of  such  errors  are 
enormous,  and  their  prevention  is  frequently  an  overriding  factor  in  the 
choice  of  system  design  and  configuration. 

For  some  systems  security  is  a  vital  factor.  What  is  the  probability 
that  a  saboteur  could  take  over  a  system  and  render  it  incapable  of  use,  or 
worse,  .use  it  against  us?  Although  it  may  be  difficult  to  quantify  both  safety 
and  security,  there  can  be  no  question  that  system  design  and  operation 
criteria  must  reflect  a  thorough  assessment  of  these  real  probabiiities.and 
due  consideration  should  be  given  to  their  inclusion  in  a  total  effectiveness 
model. 

» 

3.  Selecting  a  Figure  of  Merit 

A  cost-effectiveness  optimization  process  is  essentially  one  of  achieving 
a  combination  of  resources  and  attained  effectiveness  that  is  best  by  some 
FOM.  In  defining  an  appropriate  FOM,  one  is  faced  with  a  problem  similar 
to  that  of  stating  in  precise,  quantifiable  terms  the  rules  or  criteria  for 
choosing  the  "best"  painting  or  "best"  automobile.*  These  examples  do  have 
some  quantifiable  (though  not  necessarily  pertinent)  characterises,  such  as 
the  size  of  the  painting,  rating  of  the  artist,  or  the  dimensions  (roominess) 
of  the  automobile;  however,  artistic  judgment  and  user  experience, 
respectively,  are  also  factors  in  the  final  choice.  In  the  same  sense,  the 
choice  of  the  best  weapon  system  is  greatly  influenced  by  the  use  of  good 
engineering,  economic,  and  operational  judgment . 
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The  only  general  rule  to  be  followed  in  selecting  an  FOM  is  that  it 
should  include  as  many  system  significant  factors  as  possible  so  that  the 
optimization  process  will  reflect  a  truly  balanced  trade-off  between  alter¬ 
natives.  But  in  order  to  keep  the  optimization  problem  within  manageable 
proportions,  the  system  and  the  boundaries  must  be  explicitly  defined.  This 
will  restrict  the  choice  of  parameters  in  the  optimization  model.  The  pur¬ 
chaser  of  a  new  automobile,  for  example,  may  or  may  not  consider  the 
service  policies  of  the  manufacturer  arid  dealer'.  If  he  does,  the  system  is 
both  the  automobile  and  service  policies;  if  he  does  not,  the  system  is  only 
the  automobile.  In  attempting  to  optimize  a  weapon  system  such  as  a  bomber, 
it  is  necessary  to  consider  whether  the  system  is  to  be  defined  as  a  single 
bomber,  a  squadron  of  bombers,  or  the  complete  bomber  fleet.  It  is  possible 
that  optimizing  with  respect  to  a  single  bomber  (a  sub -optimization)  may  not 
yield  the  optimum  "squadron"  system,  which  may  not,  in  turn,  give  a  force¬ 
wide  optimum. 

A  further  restriction  in  the  size  of  the  optimization  problem  may  be 
obtained  if  some  factors  may  be  considered  fixed  by  results  of  previous 
analyses  (perhaps  sub-optimizations).  A  maintenance  trouble-shooting 
routine,  for  example,  might  normally  be  considered  as  a  variable  factor, 
but  past  analysis  in  this  area  might  be  used  to  select  a  particular  routine 
applicable  to  the  system  under  study,  or  perhaps  to  restrict  the  range  to 
several  alternatives. 

It  is  impossible  to  establish  rigid  ground  rules  or  procedures  for 
formulating  a  criterion  for  optimizing  cost-effectiveness  of  a  system 
The  answers  to  the  following  two  basic  questions,  however,  will  provide  a 
great  deal  of  insight  for  such  formulation: 

•  Why  is  the  system  being  developed? 

•  What  physical  and  economic  limitations  exist? 

The  answer  to  the  first  question  is  essentially  given  by  the  mission 
definition  for  the  system.  Where  possible,  the  mission  definition  should  be 
translated  into  quantitative  system  requirements  --a  difficult  task  in  many 
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cases.  A  performance  measure  such  as  kill -probability  for  a  bomber  may 
be  assignable,  but  the  bomber  may  also  have  a  mission  to  act  as  a  deterrent 
--a  measure  that  is  difficult  (if  not  impossible)  to  quantify.  It  is  for  this 

M 

type  of  multimission  case  that  judgment  will  become  especially  important. 
Even  if  quantitative  requirements  can  be  placed  on  all  mission  types, 
weighting  factors  will  have  to  be  introduced  to  quantify  the  relative  impor¬ 
tance  of  each  mission. 

Factors  that  have  relatively  little  impact  on  over-all  effectiveness  or 
cost  can  be  considered  to  be  fixed  —  or,  possibly,  ignored.  There  is,  of 
course,  a  risk  involved  if  factors  chosen  to  be  fixed  or  unimportant  would 
have  had  a  significant  effect  if  they  had  been  allowed  to  vary.  Factors  that 
fall  in  this  "gray  area"  may  have  constraints  imposed  upon  them  in  Such  a 
manner  that  the  more  detailed  analysis  to  be  performed  in  the  optimization 
process  will  indicate  final  disposition.  For  example,  if  a  questionable 
factor  might  have  a  monotonic  influence  on  effectiveness,  consideration  of 
only  extreme  values  might  be  all  that  is  necessary  to  determine  the  signifi¬ 
cance  of  this  influence. 

In  many  areas  cost-effectiveness  criteria  or  measures  are  commonly 
accepted.  These  are  often  stated  in  terms  of  dollars  per  unit  of  task  per¬ 
formed.  These  measures  are  analogous  to  sales  prices  for  units  of 
measureable  materials  used  in  the  civilian  market,  such  as  dollars  per 
gallon,  dollars  per  pound,  etc.  These  measures  are  easily  understood 
and  lend  themselves  to  the  spirit  of  th^tirive  to  produce  or  purchase  the 
most  for  the  least.  Table  II  lists  some  examples  of  cost-effectiveness 
criteria  and  the  field  of  endeavor  in  which  they  are  used. 


TABLE  II 
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EXAMPLES  OF  COST-EFFECTIVENESS  CRITERIA 
IN  VARIOUS  AREAS  OF  ENDEAVOR 

M 

Example  of  a 

Area  of  Endeavor  Cost-Effectiveness  Criterion* 


Non -Military: 


Building 

1  Dollars  per 

Air  passenger 

Dollars  per 

Freight 

Dollars  per 

Computer 

Dollars  per 

Communications 

Dollars  per 

Electricity 

Dollars  per 

Gas 

Dollars  per 

Public  highways 

Dollars  per 

Farming 

Dollars  per 

square  foot 
passenger  mile 
ton  mile 
bit 

message  unit 
kilowatt  hour 
cubic  foot 
mile  ■' 

acre 


Military; 


Launch  vehicles 

Satellites 

Missiles 

Interceptors 


Dollars  per  pound  payload  in 
orbit 

Dollars  per  hour  of  successful 
operation  in  orbit 
Dollars  per  kill 
Dollars  per  intercept 


Cost  per  successful  effort  is  different  for  military  than  for 
non-military  products,  since  success  is  usually  probabilistic 
in  nature  in  the  military  situations. 
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\  F.  BLOCK  5.0  SPECIFICATION  OF  ACCOUNTABLE  FACTORS 

As  a  preliminary  to  model  construction,  and  following  mission  defini¬ 
tion^  N  system  description,  and  specification  of  figures  of  merit,  it  is 
necessity  to  spell  out  the  boundary  conditions  of  the  analysis  to  be  conducted. 

First,  the  (5.  1)  Level  of  Accountability  must  be  specified. 

5.  l.vl  What  are  the  System  Interfaces?  Is  the  system  a  single 
missile,  or  is  it  a  wing?  Is  it  to  be  regarded  as  an  independent  entity,  or 

S 

is  it  to  be  considered  on  a  force  mix  basis  ? 


5.  1.2  What  is  the  Level  of  Analysis?  Is  a  missile  a  least 
unit?  a  subsystem?  a  launch  site  replaceable  module?  a  piece  part? 

5.1.3  What  are  the  variables  of  the  Analysis?  The  variables 
of  an  analysis  are  of  two  kinds: 

/  :  -•  *'  • 

•  controllable,  or 

<■  •  fixed. 


Both  are  required  to  estimate  effectiveness.  Only  the  former  is  subject  to 
trade-off.  Expected  trade-offs  should  be  identified  before  model  construc¬ 
tion  commences.  Typical  variables  are: 

•  failure  rates  by  mode  of  operation  and  by 
subsystem,  module,  or  piece  part; 


•  downtime  distributions  categorized  by  equipment 
and  reason; 

test  coverage  by  subsystem  and  test  designation- 

•  distribution  of  task  durations  by  test  designation; 

•  personnel  factors ; 

•  spares  provisioning; 

•  deployment  ; 

•  maintenance  factors. 
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It  is  also  necessary  to  (5.2)  Define  Constraints  particularly  in  the 
areas  of  •  '  (  ' 


5.2.  1 

Data 

5.2.2 

Schedule  > 

5.2.  3 

Burden 

5.2.4 

Resources 

5.2.  5 

Acceptable  risk  and  uncertainty 

5.2.6 

Physical  environment 

Those  (5.  3)  Personnel  factors  which  it  is  desired  to  see  reflected  in 
the  model  construction  and  which  have  an  impact  on  effectiveness  should  be 
carefully  spelled  out,  for  example; 

5.  3.  1  Manning  level 

5.3.2  Organization 

5.  3. >3  Characteristics 

5.  3,  3.  1  AFSC  (Skill  Level) 

5.  3.  3.  2  Task  Duration 

5.3.  3.  3  Errors  of  Commission  and  Omission 

Those  (5.4)  Procedural  factors  which  it  is  desired  to  see  reflected  in 
the  model  construction  and  which  have  impact  oh  effectiveness  should  be 
carefully  delineated,  for  example; 

5.4.1  Maintenance  Policy 

5.4.  1 .  1  Type 

5.4.  1.2  Time  Line 

5.4.2  Software 

Those  (5.5)  Hardware  factors  which  it  is  desired  to  see  reflected  in 
the  model  construction, and  which  have  an  impact  on  effectiveness,  should 
be  carefully  noted,  for  example; 


5.5.  1  System  Organization  (subsystems,  modules,  etc.) 

5.  S.  Z  Equipment  Operating  Time  Line  by  Mode  and  Equipment 

5.  5.  i  Failure  Distributions  by  Mode  and  Equipment 

The  (5.6)  Logistics  factors  and  the  (5.7)  Scenario  of  system  planned 
use  should  bo  accounted  for.  Table  III  presents  a  typical  checklist  of 
accountable  factors.  This  is,  of  course,  only  a  partial  checklist  of  account¬ 
able  factors.  It  serves  only  as  a  point  of  departure. 


G.  BLOCK  6.0  IDENTIFY  DATA  SOURCES 

The  structure  of  a  model  must  be  tailored  to  fit  the  type  of  data  available. 
This  is,  of  course,  a  two  way  road.  The  type  of  question  to  be  answered 
ami  the  type  of  trade-off  to  be  considered  imply  a  need  for  certain  types  of 
data.  The  early  identification  of  data  sources  aids  in  formulating  a  proper 
model  structure  and  alerts  management  to  include  timely  planning  for  (8.0) 
Data  Acquisition. 
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TABLE  IH 


TYPICAL  CHECKLIST  FOR 
IDENTIFIC  ATION  OF  ACCOUNTABLE  FACTORS 


System  Hardwire  Description 

Spares 

Modes  if  operation 

Provisioning 

Hardware  organization 

Storage 

Compatibility 

Packaging 

(e.g..  Electromagnetic 

Support  Equipment 

Compatibility) 

T  est 

Survivability 

T  ran sport 

Vulnerability 

Deployment 

Maintenance 

Facilities 

Procedures /Policies 

Geographic  Factors 

Operating 

•  Deployment 

Repair 

•  Geology 

Inspection /Maintenance 

•  Climate 

•  Atmospheric  phenomena 

Testing 

System  Interfaces 

Personnel 

•  Support  systems 

•  Operating 

■  Force  mix 

•  Maintenance 

•  Strategic  Integrated 

Transportation 

Operations  Plan  (SIOP) 
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H.  BLOCK  7,0  MODEL  CONSTRUCTION 

The  WSEIAC  views  model  construction  as  a  four  step  process: 

7.  1  List  Assumptions 

7.2  List  Variables  and  Define  Model  Parameters 

7.  1  Construct  Effectiveness  Model(s) 

7.4  Construct  Cost  Model(s) 

The  (7,  1)  listing  of  assumptions  is  crucial.  The  usefulness  of  a 
model  can  bo  severely  limited  if  the  assumptions  violate  reality.  A  clear 
statement  of  assumptions  is  therefore  a  necessity  in  judging  the  validity  of 
the  results  of  a  model  exercise. 

The  (7.2)  listing  variables  and  defining  the  model  parameters  per¬ 
mits  a  comparison  of  the  structure  of  the  model  with  the  list  of  accountable 
factors  (5.  )).  It  provides  a  means  of  judging  the  completeness  of  the  model 
structure. 

A  variable  is  defined  here  as  a  quantity,  the  use  of  which,  when 
varied,  will  result  in  variations  in  resources  or  the  effectiveness  with  which 
the  program  objectives  are  accomplished.  The  step  (7.2)  in  the  task 
analysis  consists  of  identifying  those  variables  which  will  influence  the 
evaluation  of  each  alternative  listed  from  (3.  1). 

Table  IV  shows  examples  of  variables  which  can  influence  the  choice 
of  alternatives.  In  a  cost-effectiveness  optimization,  it  is  evident  that, 
although  many  variables  exist  and  could  influence  final  selection  of  an  alter¬ 
native  approach,  the  variables  which  are  significant  can  be  limited  to  those 
which  have  an  impact  on  cost,  resources  available,  or  the  effectiveness 
with  which  the  system  performs  its  function.  Model  parameters  which  do 
not  influence  these  quantities  significantly  should  not  be  included  in  the 
optimization  process. 

Variables  can  be  screened  to  a  certain  extent.  In  general,  some 
variables  can  be  arbitrarily  treated  as  fixed  quantities  as  a  result  of  the 
statement  of  requirements,  limitations  on  resources,  or  other  previously 
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TABLE  IV 


TYPICAL  VARIABLES  INFLUENCING 
EFFECTIVENESS/COST-EFFECTIVENESS 
EVALUATION  OF  ALTERNATIVES 

Cost 

Weight 

Payload  Carried 
Mission  Length 
State -of -the  -  Art 
Time  Required 
Reliability 
*  Safety 

Maintenance 

Availability 

Vulnerability 

Survivability 


established  decisions  on  the  program.  In  other  cases,  a  legitimate  variable 
can  be  treated  as  a  fixed  quantity  initially.  Then,  after  initial  optimizations 
have  been  completed,  the  effect  of  altering  the„variable  can  be  expressed  in 
terms  of  impact  on  the  final  answer.  In  many  cases,  judicious  fixing  of 
variables  in  this  manner  can  save  a  large  amount  of  manpower  expenditure 
if  the  decision  to  fix  the  variable  is  based  upon  probable  insensitivities  of 
the  answer  to  the  magnitude  of  variation  expected. 

The  range  of  each  variable  to  be  considered  should,  for  economy  of 
analysis  effort,  be  limited.  Constraints  on  physical  characteristics  often 
limit  the  range  of  performance  characteristics  or  other  variables  which  can 
be  considered.  Preliminary  sensitivity  analysis,  rough-cut  analysis,  or  an 
extreme  (maximum  and  minimum)  value  analysis  are  also  useful  in  indicating 
probable  limits  of  variables.  Variables  thus  limited  should  be  re-examined 
after  completion  of  the  optimization  study.  If  a  definite  optimum  point  is 
reached  within  the  limits  of  each  variable,  it  is  generally  safe  to  assume 
that  the  limits  established  were  reasonable. 

A  model  parameter,  as  used  here,  denotes  a  specific  symbol  (or  name 
of  a  quantity)  which  enters  into  calculating  system  effectiveness /cost- 
effectiveness.  Parameters  may,  themselves,  be  variables,  or  variables 
may  be  compounded  of  parameters;  but  parameters  will  always  be  the 
smallest  identifiable  units  of  a  model.  They  define  the  fine  structure  of  a 
model.  Care  must  be  taken  to  select  parameters  which  can  be  estimated 
from  available  data  and  which  at  the  same  time  permit  the  study  of  variations 
in  independently  controllable  factors.  For  example,  the  mission  reliability 
of  a  single  unit  should  not  be  treated  as  a  lumped  value,  but  should  rather  be 
expressed  in  terms  of  the  two  independently  controllable  parameters,  the 
mission  duration  and  the  unit  failure  rate. 

•  The  WSEIAC  has  outlined  a  specific,  basic,  analytical  model  (see 
BLOCK  4.0  discussion)  on  which  to  (7.  3)  construct  effectiveness  model(s).  In 
its  symbolic  form,  effectiveness  (E)  is  given  by, 

E  =  A'  [D]  C 
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where 


E  =  System  Effectiveness  is  a  measure  of  the  extent  to  which  a 
:  system  may  be  expected  $o  achieve  a  set  of  specific  mission 

*"  requirements  and  is  a  function  of  availability,  dependability 

and  capability. 

_ 1 

A  =  Availability  is  a  measure  of  the  system  condition  at  the 
start  of  a  mission  and  is  a  function  of  the  relationships 
among  hardware,  personnel  and  procedures. 

[D]  =  Dependability  is  a  quantitative  measure  of  the  system  con¬ 

dition  at  one  or  more  points  during  the  mission,  given  the 
system  condition(s)  at  the  start  of  the  mission,  and  may  be 
stated  as  the  probability  (or  probabilities  or  other  suitable 
mission  oriented  measure)  that  the  system  will  enter  and/or 
occupy  any  one  of  its  significant  states  during  a  specified 
mission, 

^  =  Capability  is  a  measure  of  the  ability  of  a  system  to  achieve 

the  mission  objectives,  given  the  system  condition(s)  during 
the  mission,  and  specifically  accounts  for  the  performance 
spectrum  of  a  system.  „ 

The  first  step  in  implementing  this  analytical  definition  is  to  des¬ 
cribe  the  significantly  different  system  "states"  in  which  the  mission  may  be 
carried  out.  System  "states"  are  distinguishable  conditions  of  the  system 
which  result  from  events  occurring  prior  to  and  during  the  mission.  For 
example,  the  condition  in  which  all  system  hardware  is  functioning  within 
design  specifications  is  one  state,  .he  condition  in  which  the  system  is 
completely  inoperable  due  to  hardware,  personnel,  or  procedural  failures 
is  a  state  at  the  other  extreme.  The  conditions  of  partial  system  operation 
due  to  defects  of  hardware,  personnel,  or  procedures  are  represented  by 
the  intermediate  system  states.  It  should  be  evident  that  the  system  cart 
make  transitions  from  state  to  state  during  a  mission.  The  time-line  J 
analyses  performed  in  accordance  with  5.3,  5.4  and  5,5  may  have  split 
the  mission  into  a  number  of  discrete  time  intervals  during  which  different 
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functions  are  being  performed  and  different  portions  of  the  system's  hard¬ 
ware  are  being  used.  For  each  discrete  time  interval,  a  set  of  significant 
states  appropriate  to  the  function  being  performed  during  that  interval  must 
be  defined.  * 

The  next  step  is  to  relate  probabilities  to  each  of  the  sets  of  signifi¬ 
cant  states  which  arc  appropriate  at  the  beginning  of  the  mission.  This 
array  of  probabilities  is  called  the  avci’ability  vector.  For  each  succeeding 
time  interval,  an  array  of  state  probabilities  is  related  to  accountable  fac¬ 
tors.  These  probabilities  .arc  dependent  or  conditional  on  the  effective  state 
during  the  previous  time  interval.  For  example,  where  no  repair  is  possi¬ 
ble,  a  failure  in  one  interval  predetermines  the  possible  states  in  the 
■'succeeding  intervals.  These  arrays  of  conditional  probabilities  ar e^called 
the  dependability  matrices. 

A  simplified  method  of  analysis  which  is  generally  employed,  defines 
the  significantly  different  (effective)  system  states  over  the  entire  mission 
rather  than  for  each  discrete  time  interval.  The  array  of  state  probabili¬ 
ties  at  the  beginning  of  the  mission  still  yields  the  availability  vector. 
However,  the  dependability  matrix  contains  the  probabilities  of  the  effective 
states  throughout  the  mission  conditional  on  the  initial  states. 

The:  last  step  is  the  construction  of  the  capability  vector.  This  is 
an  array  of  numbers  which  arc  a  measure  of  the  ability  of  a  system  to 
achieve  the  mission  objectives;  given  the  system  condition(s)  during  the 
mission.  This  array  of  numbers  (vector  or  matrix)  Specifically  accounts 
for  the  performance  spectrum  of  the  system.  A  spectrum  of  possible 
mission  results  occurs,  for  example,  when  the  accumulation  of  subsystem 
performance  deviations,  each  within  acceptable  tolerances,  results'  in  a 
bomb  drop  being  wide  of  the  mark.  In  this  case,  there  has  been  no  specific 
subsystem  malfunction,  but  a  system  malfunction  (or  performance  degrada¬ 
tion)  due  to  the  unlikely  combination  of  within  tolerance  variations  of  the 
subsystem.  There  may,  therefore,  be  a  continuous  spectrum  of  possible 
mission  results,  none  of  which  is  an  unequivocal  failure  or  success.  The 
capability  matrix  represents  the  "worth"  or  "value"  of  each  system  state. 
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Each  element  of  the  matrix  is  the  mission  worth  which  accrues  from  ; 
carrying  out  the  mission  in  a  given  effective  state. 


The  basic  analytical  framework  given  above  is  not  intended  to  be 

restrictive.  This  point  is  illustrated  in  the  radar  detection  and  tracking 

example  of  Volume  II  of  the  Task  Group  II  report  where  the  following  yaria- 

(3) 

tions  on  the  basic  model  arc  illustrated: 


Ej  =  A  C(0) 

E2  =  X  LC(0)]  [D(30)]  C(30) 


E 


3 


E2 


.  In  the  first  variation,  the  system  effectiveness  (Ej)  is  defined  to  be 
the  probability  that  the  radar  will  adequately  perform  initial  detection  of 
the  target.  In  this  case  the  dependability  matrix  reduces  to  unity  since 
"mission  duration"  is  measured  from  the  point  of  initial  detection,  and  CT 
applies  to  detection  capability  only  (denoted  by  (^(0)).  In  the  second  varia¬ 
tion,  the  system  effectiveness  (E^)  is  defined  to  be  the  probability  of  initial 
detection  and  track  for  a  period  of  thirty  miniutes.  In  this  case,  the  ele¬ 
ments  of  the  detection  capability  vector  C(0)  become  the  elements  of  a 
capability  matrix  [C{0)]  are  now  combined  with  a  dependability  matrix 
lD(30)]  and  a  new  capability  vector  C(30)  which  express  the  tracking 
capability  of  the  radar  for  a  period  of  thirty  minutes.  In  the  final  variation, 
the  system  effectiveness  (E^)  is  defined  to  be  the  probability  of  successful 
track,  given  initial  detection.  This  conditional  measure  is  the  ratio  of  the 
two  previously  treated  variations. 

The  intended  flexibility  of  approach  is  further  illustrated  in  the 

avionics  example,  which  is  Example  A  of  Volume  III  of  the  Task  Group  II 

report,  where  the  following  scries  of  effectiveness  measures  are 
(41 

illustrated,' 
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The  first  measure  e! ^  treats  the  effectiveness  of  the  j**1  system  function 
or  subsystem  in  the  i*k  mode  of  operation  in  terms  of  the  basic  analytical 
model.  The  system  effectiveness  in  the  i**1  mode  of  operation  (E^l)  is 

(i)  i 

then  treated  as  the  continued  product  of  the  E^  over  the  k  subsystems 
(or  functions)  that  collectively  define  the  avionics  system.  Finally,  the  net 
effectiveness  of  the  entire  avionics  system  (E)  is  the  sum  of  the  effective¬ 
ness  of  the  system  in  each  of  its  modes  of  operation  multiplied  by  the 

probability  P.  of  utilizing  that  mode  of  system  operation,  where  m  is 
the  number  of  modes  of  operation. 


The  common  elements  in  these  variations  are  availability,  dependa¬ 
bility  and  capability.  The  precise  manner  in  which  they  combine  depends 
wholly  upon  the  specific  definition  of  system  effectiveness  which  is  to  be 
considered. 


Furthermore,  the  recommended  basic  expression  is  not  intended  to 
exclude  simulation  from  consideration.  It  is  generally  acknowledged  that 
the  complexity  of  modern  systems  will  frequently  preclude  a  detailed  pencil 
and  paper  treatment  except  as  a  first  cut  analysis.  However,  when  simula¬ 
tion  must  be  resorted  to,  it  is  recommended  that  the  computer  representation 
of  the  system  should  produce  intermediate  by-products  which  can  be  clearly 
identified  as  availability,  dependability  and  capability. 

The  value  of  (7.4)  construction  of  cost-effectiveness  models  lies  in 
the  ability  to  use  the  model  to  evaluate  new  concepts,  to  direct  effort  toward 
optimum  systems,  to  evaluate  the  effect  of  enemy  advances  in  technology, 
and  to  define  meaningful  research  and  development  programs. 
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Since  we  are  concerned  here  with  quantitative  methods,  each 
system  characteristic  or  variable  must  be  represented  by  numerical  measures. 
The  questions  of  precision,  accuracy,  qnd  consistency  of  measurement  must 
be  carefully  examined, satisfactorily  resolved,  and  presented  in  each  study. 

The  sub-models  to  be  generated  within  an  operational  concept  will  not  be  inde¬ 
pendent;  so  the  elements  jointly  involved  will  have  to  be  examined  for  con¬ 
sistency  from  sub-model  to  sub-model. 

i 

The  first  step  in  the  basic  optimization  process  is  to  relate 
the  variables  used  with  each  other  and  with  the  resources  which  are  affected. 
These  relationships  must  be  expressed  in  such  a  manner  that  the  variables 
and  resources  can  be  expressed  in  terms  of  the  established  cost-effective¬ 
ness  criteria  and  measures.  Development  of  relationships  must  be*carried 
to  the  point  where  all  resources  and  variables  can  be  related  to  either  a 
single  common  denominator,  (usually  dollars),  or  to  a  cost  denominator 
(dollars)  and  an  effectiveness  measure.  The  relationships  so  developed 
are  then  expressed  in  model  form  which  is  essentially  a  mathematical, 
logical,  or  physical  representation  of  the  interdependencies  between  the 
variables,  resources,  and  measures  of  effectiveness. 

The  WSEIAC  reports  discuss  three  basic  types  of  cost  models; 

* 

•  profit  model 

•  cost-effectiveness  ratio  model 

•  long  term  effectiveness  ratio  model. 

The  profit  model  is  simply  the  application  of  the  commercial 
concept  of  maximizing  return  on  investment.  This  may  be  stated  in  terms 
of  maximizing  absolute  return 

P  =  E  -  C 

=  value  received  (or  expected) 

-  cost  expended  (or  expected) 
or  in  terms  of  maximizing  rate  of  return 
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The  usefulness  of  either  of  these  profit  models  is  contingent 
on  solution  of  the  rather  difficult  problem  of  finding  a  common  unit  of 
measure  for  E  and  C.  This  has  been  done  in  the  past  by  such  arbitrary 
means  as  indexing  each  on  a  common  scale  (e.g.  ,  0  to  100)  or  by  relating 
E  to  value  of  targets  killed,  value  of  property  defended,  or  protected,  etc. 
However,  such  arbitrary  scaling  --  whatever  the  logic  upon  which  it  is  based 
often  leads  to  gross  misunderstanding^  and  frustrations  on  the  part  of  those 
involved  in  the  decision  making  process. 

The  rate  of  return  profit  model  suffers  from  an  additional 
difficulty.  Under  some  circumstances  the  optimum  (maximization)  of  a 
ratio  function  may  occur  at  the  origin.  That  is,  one  finds  that  the  best 
system  is  no  system  at  all! 


The  cost-effectiveness  ratio  model,  as  indicated  by  the  name, 


is  given  by: 


fj  =  C/E 


cost  expended  (or  expected) 
value  received  (or  expected) 


or 

f2  =  E/C  • 

Here  it  is  desired  to  minimize  f  j  or  maximize  f^. 


This  type  of  model  has  the  advantage  of  providing  a  cost- 
effectiveness  measure  in  natural  terms.  Thus,  in  terms  analogous  to 
transportation  (cents  per  ton-mile,  etc.),  cost-effectiveness  measures 
dollars  per  kill,  etc.  This  type  of  model  is,  therefore,  very  useful  in 
comparing  alternative  solutions  to  the  same  problem.  On  the  other,  hand, 
we  are  again  faced  with  the  dilemma  posed  by  ratio  functions.  One  way  out 
of  this  difficulty  is  to  express  fj  or  in  an  equivalent  form  using 
Lagrangian  multipliers.  Thus,  we  may  seek  tp/maximize  E  subject  to  a 
constraint  on  cost  C. 

For  example,  we  attempt  to  maximize, 

E  +  \c  (C  -CQ) 

where  CQ  is  a  fixed  budget  and  is  called  a  Lagrangian  multiplier. 
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'  V 

Schedule  may  likewise  be  accounted  for  by  means  of 
Lagrangian  multipliers .  For  example,  during  acquisition,  we  may 
maximize  > 

•  E  +  Xc  (C  -  CQ)  +  Xt  (td  -  tg) 

where 

t,  =  time  required  to  develop  a  given  level  of  E  at 
a  given  cost  C 

t  =  fixed  date  of  termination  of  development 
s  (constraint) 

Cq  =  fixed  development  budget  (constraint). 

Alternatively,  we  may  seek  to  minimize  cost  C  subject  to  a  constraint  on 
E.  That  is,  we  attempt  to  minimize  the  expression 

c  +  *e(e-eo> 

where  EQ  may  be  interpreted  as  the  minimum  acceptable  system  effective¬ 
ness  (constraint). 

It  should  be  noted  that  this  approach  does  not  require  that  the 
constraints  be  single  valued.  They  maybe  stated  as  inequalities. 

The  long  term  cost-effectiveness  model  differs  from  the 
above  cost-effectiveness  ratio  model  only  in  that  effectiveness  is  to  be 
averaged  over  the  entire  life  of  the  system.  In  general, 

t 

E  =  T--  f.  E[t]h[t]dt 

t0  "  e  *0 

date  of  initial  deployment  of  system 
date  of  system  phaseout 

the  worth  of  a  given  E  at  any  point  in  time 

between  tn  and  t  . 

0  e 

the  effectiveness  of  the  system  at  time  t. 


where 


t  = 
e 

h[t]  = 
E  = 
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This  model  is  subject  to  the  same  difficulties  as  the  foregoing  model,  and 
in  addition,  it  has  the  additional  difficulties  of  requiring  a  "judgment  function" 
h(t)  and  a  knowledge  of  the  future  E(t). 

Clearly  each  of  the  above  model  types  has  its  advantages  and 
disadvantages.  None  is  the  perfect  answer  for  all  system  evaluations.  Each 
is  useful  on  occasion.  The  choice  of  a  particular  model  will  depend  upon  the 
system  and  the  type  of  question  being  asked.  However,  whatever  the  basic 
model  type  chosen,  the,formulaiion  of  a  cost-effectiveness  model  should  be 
in  terms  of  a  maximization  or  minimization  process  subject  to  constraints. 
This  process  is  called  "trade-off"  studies  in  the  common  parlance  of  the 
industry.  Trade-offs  are  those  compromises  made  possible  by  a  substitution 
among  elements,  values,  or  materials  to  secure  a  preferred  value  of  a 
critical  system  characteristic. 

'  Table  V  lists  the  areas  which  an  analyst  must  consider  in  . 

going  from  physical  and  engineering  performance  characteristics  to  a 
relationship  between  cost  factors  and  effectiveness  factors.  ' 

A  typical  cost  estimating  relationship  is  shown  in  Figure  7. 
Estimating  relationships  within  a  known  technology  or  over  a  range  qf 
technologies  is  obtained  by  interpolating  among  data  points  developed  during 
current  or  previous  programs.  However,  if  it  is  necessary  to  extrapolate 
to  new  performance  levels,  then  the  performance  must  be  developed  by 
technology  and  then  related  to  cost.  Figure  7  shows  the  effect  of  technolo¬ 
gical  changes  on  performance -cost  relationships  and  indicates  that  empirical 
data  will  describe  the  envelope  of  the  relation. 

Effectiveness  parameters  will  be  estimable  from  some  set  of 
system  events.  The  identity  of  these  events  will  be  uniquely  defined  by  9.  1, 
Specification  of  Parameter  Estimation  Methods,  in  conjunction  With  the  8.2, 
Specification  of  Test  Methodology. 


TABLE  V 

POTENTIAL  TRADE-OFF  AREAS 


resources 

VARIABLES 

ALTERNATIVES 

-Funds  Available 

-Cost 

-Basic  Concept 

-Time  Available 

-Weight  . 

.  Manned  Versus  Unmanned 

-Payload  Capability 

-Payload  Carried 

■  Liquid  Versus  Solid 
Rockets 

-Manpower  and  Skills 

-Mission  Length 

-State  of-the-Art 

-Time  Required 
-Reliability 

-Safety 

-Maintenance 

-System  and  Subsystem  Type 

.  Battery  Power  Versus 
Generation 

•  Materials  Choice 

-System  and  Subsystem 

-Availability 

-Vulnerability 

-Survivability 

Configuration 

.  Redundancy 

•  Maintainability 

•  Hi -Reliability  Versus 

MIL  Standard  Parts 

-Operational  Modes 
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FIGURE  7.  COST  ESTIMATING  RELATIONS  BETWEEN  A 
PERFORMANCE  VARIABLE  AND  COST 
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I.  BLOCK  8.0  DATA  ACQUISITION 

Planning  for  data  acquisition  requires  careful  attention  to: 

8.  1  Specification  of  Data  Elements 

8.2  Specification  of  Test  Methodology 

8.  3  Specification  of  a  Data  Collection  System 

There  is  no  mystery  attached  to  the  8.  1  Specification  of  Data  Elements. 

Effectiveness  data  elements  (8.  1.  1)  must  be  chosen  such  that  they  determine: 

•  the  location  of  a  system-significant  event  in  space  and  time 
in  such  a  way  that  the  event  can  be  uniquely  related  to: 

>  *  concurrent  events 

•  immediately  previous  events.  * 

The  minimum  information  which  is  required  to  uniquely  specify  an 

:  i' " 

event  is: 

•  *  geographic  location  of  the  event  (site  number,  base,  depot) 

•  system  location  of  the  event  (aircraft  number,  guidance 
subsystem,  module,  etc.) 

•  name  of  the  event  (checkout,  change  of  status,  problem 
encountered,  apparent  failure,  flight,  etc.) 

•  name  of  affected  item  (part  number,  drawer  number, 
component  type,  etc.) 

•  time  of  the  event  (clock  time  and  date) 

•  action  taken  (replace  part  number  _ ,  repaired  in  place, 

performed  checkout  ivith  TO  number _ ,  etc. ) 

•  results  of  action  taken  (successful  launch,  verification  test, 
flight  aborted,  launch  delayed  "T"  minutes,  etc.) 
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The  key  to  an  adequate  data  acquisition  program  is  the  determination  of 
those  events  which  are  system  significant.  An  event  is  only  of  significance 
if  it  contributes  to  the  evaluation  of  a  parameter  of  the  system  model.  Thus, 
the  determination  of  system -significant  events  hinges  upon  the  parameter 
list  (7.2)  of  7.0  Model  Construction. 

The  above  logic  was  applied  to  the  availability  factor  of  effectiveness  for 
an  ICBM  fleet  in  the  technical  addendum  (Volume  III)  of  the  Task  Group  II 
Final  Report.  ^  The  resulting  data  requirements  were  then  used  to  judge 
the  adequacy  of  the  official  Air  Force  data  collection  system  (AFM  66-1), 
the  SAC  U-82,  and  the  SAC  U-86  data  forms  as  illustrated  in  Table  VI. 

They  flunk. 

The  cost  of  the  AFM  66-1,  Maintenance  Data  Collection  System  (MDGS) 
is  about  6.6  million  dollars  per  year,  as  a  conservative  minimum.  Surely, 
such  a  costly  system  should  yield  much  more  useful  and  more  accurate 
information. 

Clearly,  some  system  such  as  the  AFM  66-1  MDCS  should  be  used  to 
supply  data  for  system  effectiveness  measurements.  However,  the  AFM  66-1 
MDCS,  itself,  is  totally  inadequate.  This  has  been  well  documented  (refer¬ 
ence  Task  Group  n,^»3,  4)  GroUp  ni,^  Klug  Report,  Parcel  E). 

The  data  collection  forms  of  this  system  were  originally  designed  to 
obtain  maintenance  data,  not  reliability  or  other  effectiveness  data.  The  re¬ 
cent  attempts  to  use  these  forms  for  reliability  and  maintainability  have 
failed.  Recommended  changes  to  the  system,  requested  as  early  as  1961, 
arising  from  studies  such  as  that  at  Oxnard  Air  Force  Base,  California  , 
(documented  in  RAND  Memorandum  RM-3370-PR)  have  not  be  implemented. 

In  addition  to  having  insufficient  information,  the  data  forms,  them¬ 
selves,  are  so  poorly  designed  they  encourage  the  generation  of  inaccuracies 
--  both  in  key-punching  and  in  data-logging.  The  extent  of  these  inaccuracies 
is  unknown,  but  certain  contractor  studies  show  that  they  are  on  the  order  of 
ten  per  cent  to  forty  per  cent.  Decisions  based  on  such  data  should  certainly 
be  suspect.  (It  is  pertinent  to  mention  here  that  the  current  AFLC  "error 
audits"  and  "error"  percentages  do  not  reflect  data  accuracy  but,  rather, 
block-entry  accuracy.) 
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TABLE  VI 

DATA  AVAILABLE  FROM  CURRENT  AF  DATA  REPORTING  SYSTEMS 


Items  of  Informations 

U-821 2'  6 

U-863 4 5’  7 

AFM  66-1 

Location  (by  site  number  and  base) 

yes 

yes 

yes 

Name  of  checkout 

no* 

yes 

no8 9 10 11 

Name  of  subsystem 

yes^ 

yes 

yes^>  9 

Time  and  date  of  assignment  to  EWO 

yes 

no 

no 

Time  and  date  of  entry  to  checkout 

yes 

yes 

date  of 
completion 
only 

Time  and  date  of  each  problem 
encountered  in  checkout 

yes 

yes 

problem  j  ( 
only 

Description  of  each  problem 
encountered  in  checkout 

yes12 

yes 

yes  * 0 

Date  of  bench  test  of  rejected  parts 

no 

no 

date  only5 

Results  of  bench  test 

no 

no 

yes  ^ 

Date  of  tear -down  failure  ^ 
analysis  of  rejected  parts 

no 

no 

no 

Results  of  failure  analysis  ^ 1 

no 

no 

no 

1  "by  exception"  reporting;  i,e.,  only  when  condition  takes  site  off  alter 

2  was  removed  from  data  system  recently  (November  1963),  Is  scheduled 
for  return  to  data  system  when  checkout  SGC  are  detailed  in  -06  code  books 

3  reports  countdown  only 

4  through  Work  Unit  Code  correlation  only 

5  available  for  recoverable  items  only 

6  key  punched  for  machine  processing 

7  not  keypunched,  or  only  partially  keypunched 

8  requires  Support  General  Code  of  -06  Code  and  changes  to  TO-0020E-1 

9  no  Work  Unit  Code  when  checkout  only 

10  cannot  correlate  checkout  AF  TO  Forms  and  resulting  maintenance 
problem 

11  can  be  directed  by  responsible  AMA  as  a  special  task  for  problem  areas 

12  problem  --  frequently  cannot  be  correlated  to  checkout  data. 
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The  AFM  66-1  MDCS  needs  major  changes;  i.e.,  formats,  key  punching, 
method  of  tape  record  storage,  response  time,  product  outputs,  feedback  to 
bases,  accuracy  checks,  checks  on  data  dutput  product  useage.  Perhaps 
only  the  concept  itself  should  be  retained;  i.e.,  base  maintenance  logging 
data,  base  comptroller  key-punching,  forward  to  control  agency,  feedback 
to  agencies  needing  such  data. 

These  various  problems  coupled  with  the  inflexibility  to  change  of  this 
official  Air  Force  data  system  have  caused  many  of  the  commands  to  esta¬ 
blish  separate  data  systems  and  forms  piecemeal  and  for  specialized 
purposes;  e.  g.,  SAC's  U-82,  ADC's  proposed  forms,  AFSC's  Form  258-5, 
and  the  like.  The  result  today  is  that  data,  especially  data  relating  to, 
system  effectiveness,  is  currently  fragmented  throughout  the  Air  Force.  No 
agency  oversees  the  entire  effort.  In  fact,  the  absence  of  a  responsible 
agency  to  discharge  this  responsibility  seriously  hampers  data  collection 
efforts  and  wastes  money.  There  is  no  means  to  resolve  differences  among 
commands,  to  see  that  data  requested  by  one  command  and  generated  by 
another  is,  in  fact,  even  used;  no  means  to  see  that  feedback  to  data  collec¬ 
tors,  especially  on  errors,  is  heeded  and  used  to  make  the  system  more 
accurate;  no  means  to  see  that  valid  data  requests  are  processed  and^data 
given  to  agencies  who  need  it  to  make  the  system  more  effective  or  to 
determine  if  it  meets  minimum  acceptable  requirements. 

In  addition  to  effectiveness  data,  it  is  necessary  to  obtain  cost  data  if 
cost-effcctiveness  calculations /optimizations  are  to  be  accomplished. 
WSEIAC's  remarks  in  thib  area  are  limited  to  pointing  out  that  the  8.  1.  2 
specification  of  cost  data  elements  in  the  Air  Force  is  currently  not  carried 
out  with  sufficient  uniformity  nor  in  sufficient  detail.  There  must  be  uni¬ 
formity  in  the  major  data  categories,  and  the  data  must  be  kept  in  basic 
units  so  that  it  can  be  employed  in  analyses  which  require  different  view¬ 
points  and  constraints.  Figures  8,  9  and  10  present  the  cost  categories 
which  must  be  considered  in  a  cost-effcctiveness  analysis  if  it  is  to  have 
the  value  desired. 

It  is  abundantly  clear  that  data  collection  in  the  Air  Force  is  in  need  of 
substantial  improvement  if  a  system  effectiveness /cost -effectiveness 
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FIGURE  8 

COST  BREAKOUT  CHART  FOR  ESTIMATING  CRITICAL  RESOURCE  UNITS 
REQUIRED  TO  DESIGN,  DEVELOP  AND  OPERATE  A  SYSTEM 
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(Indirect  Operating  Costs) 
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FIGURE  10.  ASPECTS  OF  DIRECT  OPERATING  COSTS 


program  is  to  be  implemented.  Block  8.  3  Specify  Data  Collection  System 
of  Figure  13  indicates  those  considerations  which  should  receive  attention  in 
developing  an  adequate  system  cost  and  effectiveness  data  collection  system. 
Briefly,  they  are: 

M 

8.  3. 1  specify  administration 

8.  3.  2  specify  personnel 

8.3.3  specify  software 

8.3.4  specify  data  preprocessing  •— 

8.3.5  specify  data  transmission*- 

The  word  "data"  is  used  in  the  8.3  Specification  of  a  Data  Collection  System 
in  a  broad  context  connoting  a  wide  spectrum  of  information.  It  refers  to 
any  information,  pertaining  to  the  system,  which  is  to  be  recorded  and/or 
published.  Data  includes:  v 

Training  Manuals 
Program  Plans 

•  Management  Summaries 

•  Cost  Data 

•  Performance  Data 

•  Information  on  Research  Costs,  Facilities  Cost 

•  Engineering  Drawings 

•  Maintenance  Data 

•  Reliability  Laboratory  Data 

•  Test  Data 

'  Progress  Reports 

•  Operating  Instructions 

A  data  system  is  an  organized  methodology  or  process  used  to  gather, 
store,  retrieve,  display,  publish,  and  distribute  the  above  data.  For  such  a 
system  to  exist,  it  is  necessary  that  an  organization  exist  which  is  respon¬ 
sible  for  performing  these  functions,  that  that  organization  be  manned, 
funded,  and  have  sufficient  equipment  and  authority  to  discharge  its  respon¬ 
sibilities.  To  accomplish  its  functions  it  requires  a  mechanized 
(computerized)  system  for  data  processing. 
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A  system  effectiveness  data  system  is  then,  a  system  used  to  collect 
the  data  required  to  predict,  measure,  and  evaluate  system  effectiveness. 
Clearly,  it  is  concerned  with  cost  data,  reliability  data,  and  maintenance 
data,  as  well  as  with  other  types  of  data.  The  data  requirements  are 

if 

covered  more  thoroughly  in  Task  Group  II,  Task  Group  III,  and  Task 
Group  IV s  reports,  but  still  need  refinement  and  coordination  before  they 
can  be  programmed  into  a  mechanized  system. 


J.  BLOCK  9.0 


DATA  PROCESSING 


The  processing  of  data  for  purposes  of  providing  effectiveness  and 
cost-effectiveness  calculations  can  be  adarge  undertaking.  Attention  must 
be  given  to: 

9.  1  Specification  of  Parameter  Estimation  Methods 
9.2  Specification  of  Administration 
9.  3  Specification  of  Personnel 
9.4  Specification  of  Hardware 
9.  5  Specification  of  Software 

The  9.  1,  Specification  of  Parameter  Estimation  Methods,  is  a.  crucial 
step  involving 

9.  1.  1  specification  of  effectiveness  parameter 

estimation  methods 

. 

9.  1.2  specification  of  cost  estimation  relationships 

"Parameter  estimation"  is  defined  here  to  mean  the  specific  analytical 
techniques  used  to  reduce  raw  data.  The  specific  methods  used  depend 
upon: 

•  the  nature  of  the  quantity  being  estimated 

•  the  control  which  can  be  exerted  over  the  physical 
mechanisms  which  generate  the  data 

•  the  format  of  data  collection. 

The  9.  1.  1  specification  of  effectiveness  parameter  estimation  methods 
is  simplest  when  a  control  population  is  available.  The  literature  covers 
this  situation  quite  thoroughly.  On  the  other  hand,  when  data  are  fortuitously 
collected  under  a  variety  of  environments  (as  is  the  usual  case  with  field¬ 
generated  data)  specifying  suitable  parameter  estimation  methods  is  more 

(4) 

difficult.  This  problem  was  given  some  consideration  by  Task  Group  IIV 
but  much  remains  to  be  done  in  this  area. 
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Section  V,  of  Volume  II  of  the  Task  Group  IV  final  report  considers  the 

(7) 

9.  1.2  specification  of  cost  estimating  relationships  in  some  detail.  A  par¬ 
ticular  methodology  is  outlined  and  the  current  AFSC  format  for  recording 
cost  estimating  relationships  is  given. 

The  areas  of  9.2.  Administration;  9.4,  Hardware;  and  9.5,  Software, 

(5) 

are  treated  extensively  in  the  Task  Group  III  final  report. 


K.  BLOCK  10.0  SPECIFY  SCHEDULE 

Schedule  is  viewed  as  a  constraint  (requirement).  The  manner  in 
which  it  can  be  explicitly  accounted  for  was  discussed  in  7.0  Model 
Construction. 

L.  BLOCK  11.0  MODEL  EXERCISE 

There  are  two  principal  uses  of  models: 

•  evaluation 

•  prediction. 

Evaluation  provides: 

•  surveillance  of  current  system  status  against  quantitative 
system  requirements 

•  feedback  upon  the  efficacy  of  the  management  decision  and 
program  control  process 

•  a  means-of  determining  system  weaknesses  or  potential 
problem  areas 

•  a  point  estimate  of  system  effectiveness  which  includes  all 
pertinent  factors  within  a  uniform  framework. 

Prediction  provides  decision  aids  through: 

•  comparative  (cost-effective)  prediction/evaluation  of 
competing 

•  system  configurations 

•  problem  solutions 

calculations  of  the  effects  of  risk  and  uncertainty  expressed  as 

•  confidence  levels 

•  paramtrer  variation  studies 

•  changing  requirements  analysis. 

The  use  of  a  system  model  involves  eight  steps: 

11. 1  perform  model  checks 

11.2  calculate  FOM's 
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SxsJZrjsz 


11.3 

do  trade- 

offs  within  constraints 

11.4 

compare 

calculations  with  standard  of  reference 

11.5 

calculate 

parameter  sensitivity  curves 

11.6 

calculate 

risk 

11.7 

calculate 

effect  of  uncertainty 

11.8 

interpret 

runs. 

The  purpose  o£  11.1  Perform  Model  Checks  is  to  test  the  basic  struc¬ 
ture  of  the  model. 

This  consists  of  a  set  of  checks  on: 

.  assumptions 

•  adequacy 

•  representativeness 

•  risk  and  uncertainty 

•  validity 

Assumptions  All  assumptions  required  for  the  model  should  be  ex¬ 
plicitly  stated  and,  if  possible,  supported  by  factual  evidence.  If  no  such 
evidence  exists,  it  is  advisable  to  state  the  reason  for  the  assumption  (e.g., 
mathematical  expediency)  in  order  to  indicate  the  degree  to  which  the 
assumptions  will  require  further  justification,  and  to  pinpoint  the  areas  in 
which  errors  might  be  introduced. 

Adequacy  A  model  must  be  adequate  in  the  sense  that  all  major 
variables  to  which  the  solution  is  sensitive  are  quantitatively  considered. 
Many  of  these  variables  will  have  been  preselected.  Through  manipulation 
of  the  model,  some  of  the  variables  may  be  excluded  or  restricted,  and 
others  may  be  introduced. 

Representativeness  Although  no  model  can  completely  duplicate  the 
"real  world,"  it  is  required  that  the  model  reasonably  represent  the  true 
situation.  For  complex  problems,  this  may  be  possible  only  for  sub-parts 
of  the  problem,  which  must  be  pieced  together  through  appropriate  modeling 
techniques.  As  an  example,  analytic  representation  may  be  possible  for 
various  phases  of  a  complex  maintenance  activity.  The  outputs  from  these 
analyses  may  then  be  used  as  inputs  to  a  simulation  procedure  for  modeling 
the  complete  maintenance  process. 
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Risk  and  Uncertainty  The  various  types  of  unknowns  involved  in 
tho  problem  cannot  be;  ignored,  nor  can  they  be  "assumed"  out;  they  must  be 
faced  squarely.  There  may  be  technological  uncertainties  involved  with 
some  of  the  system  alternatives,  operating  uncertainties  involved  with 
planning  and  carrying  out  the  mission,  uncertainties  about  enemy  strategy 
and  action,  and  statistical  likelihoods  governed  by  the  laws  of  chance  (re¬ 
ferred  to  as  risk).  The  simplest  approach  on  uncertainties  is  to  make  "best 
guesses,"  but  this  may  lead  to  disastrous  results,  since  the  probability  of 
guessing  correctly  for  every  uncertainty  is  quite  small.  For  cases  involving 
statistical  likelihood,  functions -of-random-variables  theory  or  such  proce¬ 
dures  as  Monte  Carlo  techniques  may  be  used.  For  the  other  types  of 
uncertainties,  the  general  approach  is  to  examine  all  major  contingencies 
and  compute  resultant  cost-effectiveness  parameters. 

Validity  It  must  be  recognized  that  models  will  not  be  exact 
replicas  of  the  "real  world."  Accordingly,  they  should  not  be  used  blindly. 
Portions  of  every  model  are  usually  common  to  previously  used  models  or 
can  be  related  to  quantitative  knowledge  of  trends  available  from  past 
experience.  The  model  is  validated  by  checks  in  as  many  familiar  regions 
as  possible.  The  model  is  also  checked  for  sensitivity  of  its  output  to 
changes  in  its  basic  structure.  These  sensitivity  checks  are  made  in  all 
areas  where  simplifications  have  been  made  from  the  "real  world"  care  or 
where  anomalies  have  resulted  from  the  validation  checks. 

Certain  questions  will  disclose  weaknesses  that  can  be  corrected: 

•  Consistency  -  are  results  consistent  when  major  parameters 

are  varied,  especially  to  extremes  ? 

•  Sensitivity  -  do  input -variable  changes  result  in  output 

changes  that  are  consistent  with  expectations? 


Plausibility  -  are  results  plausible  for  special  cases  where 
prior  information  exists? 

Criticality  -  do  minor  changes  in  assumptions  result  in 
major  changes  in  the  results? 
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Workability  -  «  f *»  rnndr-d  require  inputs  or  computational 

capabilities  that  are  not  available  within  the 
bounds  of  current  technology? 

•  Suitability  -  is  the  model  consistent  with  the  objectives;  i.e. 

will  it  answer  the  right  questions? 

Given  that  the  structure  of  the  model  has  been  verified,  the  figures  of 
merit  (FOM)  may  then  be  calculated  (11.2),  and  trade-off  studies  made  with 
in  constraints  (11.3).  The  object  of  tradc-off  studies  is  sytem  optimization. 

1.  OPTIMIZATION 

If  the  model  is  analytic,  the  technique  of  Lagrangian  multipliers  may 
be  employed.  This  technique  is  useful  when  well  defined  analytical  relation¬ 
ships  exist  among  the  variables,  and  when  the  constraints  are  expressly 
stated  as  either  single  valued  requirements  or  inequalities.  Alternative 
techniques  are  preferable  when  the  relations  among  the  variables  are  em¬ 
pirical,  discontinuous  or  discrete.  For  example,  when  a  finite  number  of 
discrete  alternatives  exist,  optimization  would  ordinarily  be  accomplished 

by  the  straight  forward  procedure  of  direct  comparison  of  the  calculated 

(7  8) 

costs  and  predicted  effectiveness  of  each  alternative.  ’  ’ 

When  the  data  is  empirical,  as  opposed  to  analytical,  graphical 
techniques  will  usually  prove  to  be  more  useful  and  are  particularly  useful 
when  the  constraints  are  given  as  a  bounding  range  of  acceptable  values. 
Evaluation  of  alternatives  in  this  case  may  be  treated  by  the  methods 
described  in  Section  VIII  of  Volume  II  of  the  Task  Group  IV  Final  Report. 

In  addition  to  the  above  techniques,  there  are  a  number  of  others 
available.  Among  the  more  common  are: 

•  marginal  analysis 

»  dynamic  programming 

•  simple  maximization 

•  Pontryagin's  maximum  principle 

•  linear  programming 

•  calculus  of  variations 
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•  method  of  steepest  ascent 

.  "niinimax  principle"  of  the  theory  of  games 

The  WSEIAC  has  not  given  an  illustration  of  all  these  techniques,  but 
has  chosen  to  limit  its  examples  to  illustrations  of  those  methods  which  are 
simple  to  grasp,  easy  to  exploit,  and  have  a  fairly  wide  application  to  reality, 
n  a  m  o  ly : 

•  exhaustion  of  feasible  alternatives 

•  graphical  techniques 

•  simple  maximization 

•  dynamic  programming 

•  Lagrangian  multipliers. 

The  ultimate  output  of  the  trade-off  analyses  is  a  system  configura¬ 
tion  with  a  certain  numerical  value  of  effectiveness.  Model  exercise  is  not 
complete  until  this  value  has  been  compared  to  the  quantitative  system  require¬ 
ments  (SOR)  or  other  standard  of  reference  (11.4);  a  parameter  variation 
analysis  (11.5)  has  been  made;  risk  (11.6)  and  uncertainty  (11.7)  have  been 
calculated  and  an  interpretation  of  the  results  (11.8)  has  been  accomplished. 

2.  PARAMETER  VARIATION  ANALYSIS 

The  object  of  a  parameter  variation  analysis  is  to  show  the  sensiti¬ 
vity  of  system  effectiveness /cost-effectiveness  to  changes  in: 

•  system  quantitative  requirements,  and 

•  system  variables. 

In  general,  the  results  of  such  analyses  are  given  graphically.  For 
example,  quite  simple  models  may  be  used  in  investigating  the  gross  effects 
of  alterations  in  system  support  through  parameter  variation  studies.  Con¬ 
sider  a  subsystem  which  is  periodically  inspected  every  Tg  =  60  days. 

Suppose  that  the  point  estimate  of  the  availability  is  0.  57  based  upon  the 
assumption  that  test  coverage  during  the  inspections  is  complete  (\y  =  0). 

The  effect  of  having  incomplete  test  coverage  (\  0),  and  of  varying  the 

inspection  interval  constitutes  a  parameter  variation  study.  Figure  11 
illustrates  the  results  of  such  a  study  for  this  hypothetical  system.  (The 
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point  estimate  is  circled.)  Two  conclusions  may  be  drawn  from  the  graph: 

•  If  it  is  consistent  with  manning  levels,  the  inspection  period 

k  ■  , 

should  be  reduced  from  60  to  20  or  30  days;  given  that  test 
coverage  is  complete  (X^  =  0). 

•  If  test  coverage  is  not  complete  (X^  =£  0),  resulting  in  as  ' 
little. as  one  undetectable  failure  every  100  days  (X  =  .010), 
the  proper  inspection  interval  is  quite  critical  (6-15  days) 
and  the  true  availability  cannot  exceed  0.49. 

This  analysis  suggests  two  simultaneous  actions  with  respect  to 
this  subsystem:  " 

* 

•  reduce  the  inspection  interval,  and 

•  analyze  the  test  procedures  against  the  hardware  to  gain 
.  assurance  that  X  =  0. 

K  U 

This  type  of  parameter  variation  analysis  is,  of  course,  most 
useful  during  Category  II  and  Category  III  operations. 

*  v- 

At  the  other  end  of  the  spectrum  of  parameter  variation  studies  is 
the  question  of  the  effect  of  a  proposed  improvement  on  guidance  accuracy. 

In  this  case  interest  is  in  the  change  in  unit  kill  probability  (P^).  A  general 
normalized  curve  is  shown  in  Figure  12  where  is  the  probability  of 
target  destruction,  given  impact  and  detonation  with  proper  yield  in  the 
target  area.  R^  is  the  lethal  radius  for  the  weapon  and  cr  is  ,the  standard 
deviation  of  the  accuracy  of  delivery.  The  improvement  in  may  be 

read  directly  from  this  curve  for  given  values  of  R^/ct.  For  example, 
suppose  the  current  Pk  =  0.  4  (i.e.,  R^/a  =  1),  and  the  proposed  accuracy 
improvement  is  predicted  to  yield  R^/17  =  1*75  leading  to  a  predicted 
Pk  =  0.8.  A  decision  as  to  the  worth  of  this  change  as  compared  to  buying 
more  weapons  can  be  obtained  using  Figure  13  where  we  see  that  three 
weapons  with  P^  =  0.4  equals  one  weapon  with  P^  =  0.8,  or  two  weapons 
with  -  0.8  are  worth  six  weapons  with  =  0.4. 

i.  INTERPRETATION  OF  RESULTS  ' 

During  and  after  the  accomplishment  of  the  cost-effectiveness 
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FIGURE  13.  VARIATION  OF  EXPECTED  KIEL  AS  A  FUNCTION 
OF  THE  NUMBER  OF  DELIVERED  WARHEADS 


analysis,  the  results  of  the  study  must  be  interpreted  (11,8)  in  terms' 
useful  for  the  decision  process.  Of  particular  importance  is  the  sensitivity 

,ir 

of  the  results  (i.e.,  in  terms  of  cost-effectiveness  measures)  to  variations 
in  the  input  data.  Thus,  if  the  cost-effectiveness  measure  varies  greatly 
with  some  design  parameters,  the  decision  process  must  consider  carefully 
the  uncertainty  and  the  price  paid  by  failure  to  achieve  a  design  goal. 

Interpretation  is  particularly  difficult  due  to  the  usual  communica¬ 
tion  problems  among  people  of  differing  backgrounds  and  interests.  This 
difficulty  is  amplified  by  the  nature  of  qualifying  statements  which  must  be 
made  concerning  cost-effectiveness  results  due  to  risk  and  uncertainty  and 
related  result  sensitivity.  ’ 

The  cost-effectiveness  indices  derived  from  a  given  set  of  input 
data  used  in  studies  and  models  of  the  type  described  here  are  measures  of 
"goodness"  or  "badness"  of  a  particular  system  or  system  configuration. 

The  usefulness  of  the  indices,  then,  depends  on  the  validity  of  the  model 
and  the  accuracy  of  the  input  data.  But  even  though  a  particular  submodel 
may  represent  reality  only  on  a  gross  basis,  or  a  particular  piece  of  input 
data  may  be  only  a  gross  estimate  of  reality,  the  resulting  index  may  still 
deserve  confident  consideration  as  a  measure  of  goodness.  The  key  is  the 
sensitivity  of  the  result  to  such  gross  representations.  In  sensitive  areas 
associated  with  risk  or  uncertainty,  "warning  flags"  must  be  attached  and 
some  idea  of  upper  and  lower  bounds  for  the  measure  should  be  given. 

These  warning  flags  can  be  obtained  from  parameter  variation  studies  which 
will  show  the  sensitivity  of  the  model  in  the  area  of  the  uncertainty.  Such 
sensitivity  checks  are  needed  since  the  output  of  a  cost-effectiveness 
optimization  study  is  used  to  support  program  decisions.  Sensitivity  checks 
are  intended  to  determine  the  effect  of  uncertainty  in  the  output  data  on  the 
decisions  involved.  Three  fundamental  types  of  sensitivity  checks  will  gener - 
ally  be  applicable  to  the  results  of  most  cost-effectiveness  studies.  They  are: 

•  sensitivity  to  basic  system  or  mission  requirements, 

•  sensitivity  to  uncertainties  in  estimated  or  extrapolated  data,  or 

•  validity  of  simplifying  assumptions  or  arbitrarily  fixed  variables. 
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The  basic  steps  involved  in  a  sensitivity  analysis  are: 

•  An  estimate  or  guess  should  be  made  as  to  the  possible 
numerical  range  of  uncertainty  involved.  Per  the  definition 

M 

of  uncertainty  outlined  in  the  previous  sections,  it  should  be 
recognized  that  such  estimates  or  guesses  will  generally  be 
unsupported  by  any  data  or  background  information.  They  are 
usually  a  matter  of  judgment. 

•  Using  both  the  maximum  and  minimum  values  of  the  range, 
the  optimization  analysis  or  necessary  portions  thereof, 
should  be  rerun.  The  results  of  the  nominal  and  extreme 
values  can  then  be  compared  as  required  to  determine  whether 
decisions  which  would  have  been  derived  from  the  nominal  * 
analysis  would  be  altered  if  the  extreme  values  were  believed. 

•  If  it  becomes  apparent  that  the  possible  range  of  uncertainty 
does  have  significant  effect  on  output  decisions,  steps  should 
be  taken  to  reduce  the  range  of  uncertainty,  through  either 
improvements  in  input  data  or  testing  for  more  or  better 
data,  improved  analysis  techniques  or  preparation  of  esti¬ 
mates  at  a  lower  level.  In  general,  experience  with  this  type 
of  analysis  has  shown  that  only  a  small  percentage  of  study 
inputs  involving  a  range  of  uncertainty  will  be  such  that  this 
range  of  uncertainty  will  influence  output  decisions. 

As  an  alternative,  there  are  some  situations  where  the  basic 
design  of  the  system  can  be  altered  in  such  a  manner  that  the 
system  is  no  longer  sensitive  to  the  estimated  range  of  uncer¬ 
tainties.  If  it  is  not  possible  to  remove  t,-.e  effects  of  the 
uncertainty  range  on  the  decisions  involved,  this  effect  should 
be  shown  in  visible  or  parametric  form  along  with  analysis 
results. 

If  they  are  to  be  of  value,  the  results  of  cost-effectiveness  studies  must 
be  given  in  terms  which  are  meaningful  to  those  who  make  decisions.  Thus  the 
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analysts  should  appreciate  the  problems  of  communication  with  a  broad  !tl 
spectrum  of  people  including  design  engineers,  company  managers,  military 
managers,  military  planners  and,  sometimes,  congressmen  and  the  general  - 
public. 

In  interpreting  the  results  of  these  studies,  it  must  be  remembered 
that  the  state-of-the-art,  resource  constraints,  political  and  military 
thinking  and  philosophy,  enemy  posture,  etc.,  are  in  a  constant  state  of 
flux.  Thus,  these  results  should  not  become  associated  with  hard,  fast, 
unchanging  rules.  A  current  finding  that  a  reliability  of  0.  9  is  best  for  a 
particular  component  should  not  become  permanent  dogma.  The  results 
should  never  be  the  basis  for  hindering  research.  Rather,  they  should  pro¬ 
vide  guidelines  for  further  exploration  or  tests  designed  to  yield  more 
fruitful  information  on  which  to  base  decisions. 

The  limitations  of  cost-effectiveness  studies  have  already  been 
suggested  in  the  foregoing  paragraphs.  The  reader  should  bear  in  mind  that, 
whatever  shortcomings  or  dangers  may  be  associated  with  analytical  studies 
such  as  these,  decisions  based  on  intuition,  experience  which  has  not  been 
thoroughly  analyzed,  or  a  sample  of.personal  opinions  (bone  feelings)  are 
certainly  less  defensible  and  more  subject  to  omissions  of  important  factors. 
One  would  not  build  a  bridge  by  intuitive  design,  overlooking  sound  structural 
engineering  practice;  yet  many  unknowns  exist  in  regard  to  material 
mechanics  and  random  loading  behavior  of  structures. 

Although,  in  a  sense,  statements  on  limitations  of  cost- 
effectiveness  analyses  may  be  regarded  as  platitudes,  we  present  some  of 
them  here  as  reminders. 

•  Cost-effectiveness  indices  cannot  be  meaningful  unless 
derived  from  a  model  which  represents  the  "real  world" 
fairly  closely.  Reality  should  not  be  buried  under  mountains 
of  detail  nor  does  great  detail,  by  itself,  create  reality  in 
a  model. 

It  must  be  remembered  that  cost-effectiveness  analysis  is  an 
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iterative  process.  Early  results  should  not  be  permitted  to 
create  such  a  lasting  impression  (favorable  or  unfavorable) 
as  to  lead  one  to  ignore  the  results  of  later  refinements.  This 
can  lead  to  disillusionment  on  the  part  of  all  concerned  and, 
later,  to  abandonment  of  a  valuable  tool. 

•  Cost-effectiveness  analysis  can  never  replace  good  engineering 
and  management  practice.  It  should  be  regarded  as  a  supple¬ 
mentary  tool  to  provide  meaningful  information.  Final  deci¬ 
sions  must  still  be  based  upon  sound  judgment.  This  must  be 
particularly  emphasized  since  too  many  political,  psychologi¬ 
cal  (e.  g.,  an  individual's  drive  to  solve  a  particular  problem), 
prestige  value,  and  other  factors  are  not  considered  in  a 
satisfactory  manner  at  this  time  in  such  analyses. 

•  When  results  are  sensitive  to  factors  associated  with  high 
degrees  of  risk  or  uncertainty,  "warning  signs"  must  be 
posted.  The  results  must  then  be  used  judiciously  in  making 
decisions. 

In  much  of  what  has  been  said  in  the  foregoing,  there  is  an  obvious 
attempt  to  build  up  the  importance  of  cost-effectiveness  consciousness. 
Considerable  emphasis  has  been  placed  on  developing  models  for  obtaining 
cost -effectiveness  indices  and  optimization  thereof.  However,  it  must  be 
remembered  that  these  do  not  provide  a  final  answer.  They  do  provide 
guidelines,  but  judgment  must  still  play  a  large  part. 

Perhaps,  this  is  best  expressed  by  Dr.  Alain  Enthoven's  statement--^ 
"Do  judgment  and  experience  have  no  place  in  this  approach  to  choice  of 
weapon  systems  and  strategy  and  design  of  the  defense  programs?  Quite 
the  contrary.  The  statment  that  the  issue  is  judgment  versus  computers  is 
a  red  herring.  Ultimately  all  policies  are  made  and  all  weapon  systems  are 


■J-  From  a  lecture,  "Decision  Theory  and  Systems  Analysis,"  delivered 

during  the  Distinguised  Lecture  Scries,  sponsored  by  the  Board  of  Trade 
Science  Bureau,  Washington,  D.  C.,  December  5,  1963. 
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chosen  on  the  basis  of  judgments.  There  is  no  other  way  and  there  never 
will  be.  The  question  is  whether  those  judgments  have  to  be  made  in  the  fog 
of  inadequate  and  inaccurate  data,  unclear  and  indefinite  issues,  and  a 

it 

welter  of  conflicting  personal  opinions,  or  whether  they  can  be  made  on  the 
basis  of  adequate,  reliable  information,  relevant  experience,  and  clearly 
drawn  issues.  The  point  is  to  render  unto  computers  the  things  that  are 
computers'  and  to  judgment  the  things  that  are  judgment's.  In  the  end,  there 
is  no  question  that  analysis  is  but  an  aid  to  judgment  and  that,  as  in  the  case 
of  God  and  Caesar,  judgment  is  supreme." 

Thus,  although  there  are  limitations  in  the  modeling  process  used  to 
obtain  cost-effectiveness  indices,  it  must  be  remembered  that  this  approach 
allows  us  to: 

•  organize  and  set  into  proper  perspective  the  many  alternatives 
of  the  problem; 

•  establish  many  "if-then"  statements,  pertaining  to  the  alternatives 
of  the  problem; 

’  properly  evaluate  data  uncertainties; 

•  examine  many  cases  quickly  which  would  require  years  of 
simulated  combat  to  test;  and 

•  explore  systematically  those  cases  which  cannot  be  tested 


(you  cannot  go  to  war  to  test  system  effectiveness). 


BLOCK  12.  0  PREPARE  MANAGEMENT  SUMMARY  REPORTS 


The  12.  1  Specification  of  Content  and  12.2  Specification  of  Format 
of  management  summary  reports  has  been  given  careful  consideration  by  the 
WSEIAC.  Summary  reports  should  contain: 

•  system  quantitative  requirements, 
current  status, 

•  resources  (remaining), 

■  trends, 

•  summary  of  problem  areas, 

'  optimum  (re)  allocation  of  resources,  and 

•  risk  and  uncertainty  qualifications. 

The  format  suggested  is: 

•  trend  line  charts  by  system  and  subsystem  in  graph  form 
showing  * isk  and  uncertainty 

backed  up  by 

three  levels  of  tabular  (matrix)  detail 

and 

■  variational  studies  in  graph  form  with  risk  and  uncertainty 

5  / 

shown—  . 


I  !  k  Croup  III  Final  Report,  Appendix  III,  Exhibits  1  through  13. 
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BLOCK  13.0  DECISION  PROCESS 


The  WSEIAC  viewpoint  is  that  cost-effectiveness  prediction  is  the  focal 
point  which  provides  management  a  perspective  relationship  between  system 
status,  available  resources,  constraints,  and  system  quantitative  require¬ 
ments.  The  12.  0  Management  Summary  Reports  provide  management 

•  a  summary  of  current  (or  predicted)  status 

•  a  summary  of  current  (or  predicted)  resources 

■  a  summary  of  current  (or  predicted)  trends 

•  a  summary  of  current  (or  predicted)  problem  areas 

In  addition,  and  in  response  to  management  initiative  and  qviery,  the 
summary  reports  will  contain  5 


•  the  predicted  consequences  of  possible  alternative 
actions 

and 

a  gauge  on  the  effects  of  risk  and  uncertainty. 

Thus  the  direction  that  the  11.0  Model  Exercise  takes  and  the  con¬ 
tents  of  the  12.  0  Management  Summary  Reports  directly  reflect  the 
tentative  courses  of  action  postulated  by  management.  Not  only  does  the 
ultimate  responsibility  for  decision  rest  with  management,  but  so  also  does 
the  vital  activity  of  posing  the  proper  questions.  Thus,  cost-effectiveness 
prediction  is  a  dynamic  process  which  cannot  occur  without  active  manage¬ 
ment  participation. 


In  spite  of  the  catalytic  role  played  by  management,  the  13.0  Decision 
Process,  as  such,  has  not  been  discussed  by  the  WSEIAC.  This  is  not  an 
oversight,  but  a  recognition  of  the  lack  of  information.  Formal  effective¬ 
ness /cost-effectiveness  prediction  of  the  scope  envisioned  by  the  WSEIAC  is 
without  precedent.  Implementation  of  the  WSEIAC  recommendations  will 
certainly  have  an  impact  on  current  management  decision  processes.  De¬ 
cision  processes  will  tend  to  become  more  formalized.  The 
use  of  formal  decision  algorithms  will  become  more  wide  spread. 
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It  is  a  strong  recommendation  of  the  WSEIAC  that  a  study  be  instigated 
to:  a 

'  ..'jh  ■  '  .  ,  •!  ,  ■  :j 

•  clearly  define  the  management  uses  of  models,  and 

.  develop  decision  algorithms  consistent  with  the  WSEIAC 
effectiveness/cost-effectiveness  concepts. 

Unless  the  outputs  or  final  results  of  the  cost  optimization  study  can  be 
placed  in  a  form  suitable  for  support  of  program  decisions,  the  application 
of  optimization  principles  becomes  only  an  academic  exercise.  The  type  of 
decision  support  outputs  which  can  be  generally  derived  from  the  optimiza¬ 
tion  technique  are:  .  V 

•  definition  of  design  and  mission  associated  with  the 
optimum  point ; 

•  criteria'for  evaluation  and  decision  on  future  improve¬ 
ment  alternatives;  and 

•  parametric  data  for  use  in  studies  of  other  systems. 

The  basic  optimization  techniques  described  here  involve  incorporation 
of  alternative  approaches  up  to  the  optimum  or  to  a  resource  cut-off  point, 
whichever  occurs  first/7Wierently  this  approach  is  such  that  a  definition  of 
the  configuration,  mission  and  other  scenario  associated  with  the  optimum 
point  is  readily  determined,  thus  leading  directly  to  3.  2  Configuration 
Documentation. 

As  the  program  progresses  through  Definition  Phase  and  through  its 
operational  life,  the  description  of  alternatives  (3.  1)  involved  in  the 
optimization  process  will  become  increasingly  definitive  and,  in  turn,  the 
definition  of  the  system  corresponding  to  the  optimum  point  will  become 
more  firm  (3.2).  In  addition  to  system  configuration  definition,  there  are 
a  number  of  other  natural  by-products  of  the  optimization,  such  as:  reli¬ 
ability  and  maintainability  estimates,  required  number  of  operational  units, 

<  te.  In  short,  it  is  possible,  during  successive  iterations  of  the  cost- 
effectiveness  analysis,  to  determine  those  elements  that  significantly 
influence  effectiveness  and  cost  which  are  within  the  scope  of  the  decision- 
m. li-.ing  process. 
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It  must  be  recognizqfl  that  in  the  time  period  following  completion  of  an 
optimization  ankiysis,  further  system  improvement  alternatives  will  be  con¬ 
sidered  for  incorporation.  The  scope  and  complexity  of  an  over -all  system 
optimization  analysis  are  generally  such  that  it  is  undesirable  to  rerun  the 
entire  study  each  time  such  an  improvenaent  alternative  is  to  be  considered. 
As  a  result  then,  it  is  generally  desirable  to  derive  evaluation  models  and 
criteria  from  the  optimum  point  and  the  resource  relationships  accompanying 
this  point,  in  such  form  that  future  improvement  alternatives  can  be  evalu¬ 
ated  somewhat  independently  of  the  over -all  system  analysis.  There  is, 
admittedly,  an  implicit  danger  in  following  such  an  approach  in  that  when  a 
large  enough  number  of  such  alternatives  have  been  incorporated  into  the 
system,  the  original  optimization  analysis  becomes  invalid,  thus  admitting 
the  possibility  of  erroneous  decisions.  Determination  of  the  circumstances 
and/or  frequency  under  which  the  total  optimization  must  be  reiterated  is  a 
matter  of  individual  judgment.  General  guidelines  for  formulation  of  evalu¬ 
ation  criteria  for  future  improvements  are: 


It  is  desirable  to  issue  su-ch  criteria  in  a  form  that  can 
be  utilized  by  an  individual  designer. 

The  form  must  be  such  that  a  single  criteria  or  cut-off 
is  used.  Thus,  supporting  data  and  relationships  must 
be  provided  so  that  all  variables  and  resources  which  , 
are  involved  in  future  evaluations  can  be  related  in  terms 
associated  with  the  selected  criteria. 


BLOCK  14.0  IMPLEMENT  DECISION 


Implementation  of  decisions  resulting  from  cost-effectiveness  analyses 
is  no  different  from  implementing  a  decision  from  a  design  review,  a 
schedule  change  or  other  significant  program  event.  The  block  is  included 
because  it  is  an  essential  step  in  closing  the  loop  and  without  which  the 
model  results  and  decisions  alone  are  meaningless.  The  Task  Group  IV 
report  contains  pertinent  discussion  on  management  assurance  activities 
during  program  surveillance. 
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BLOCK  15.0  CHANGE  ANALYSIS 

The  implementation  of  a  decision  based*on  effectiveness/cost- 
effectiveness  consideration  generally  implies  a  change  in  one  or  more  of 
the  following  areas: 

•  schedule 

•  model(s) 

•  system 

•  requirements. 

Each  iteration  of  the  effectiveness/cost-effectiveness  prediction/  V' 
evaluation /augmentation  cycle  should  be  accompanied  by  a  15.0  Change 
Analysis  against  each  of  these  areas.  The  result  of  this  activity  will  be 
monitoring  of  the  net  effect  of  each  decision  and  the  accomplishment  of 
program  surveillance. 
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AN  EXAMPLE  ANALYSIS  FOR  THE  CONCEPTUAL  PHASE 

INTRODUCTION 

It  is  clear  from  the  task  outline  of  Appendix  I  and  from  the  example 
analyst  s— ^  present ed  by  the  WSEIAC  that  the  prediction  of  system  effective¬ 
ness  or  cost-effectiveness  will  generally  be  an  enormously  detailed 
undertaking  in  the  Definition  and  following  phases,  On  the  other  hand, 
during  the  Conceptual  Phase  so  little  information  is  likely  to  be  available 
that  the  analyses  will  be,  at  most,  skeletonized  versions  of  the  later 
analyses.  Because  of  that  very  paucity  of  detail,  however,  an  example  of 
the  type  of  analysis  that  might  be  conducted  in  the  Conceptual  Phase  is  a 
useful  means  of  concurrently  illustrating  the  foregoing  task  analysis  and  of 
emphasizing  the  principal  problem  areaB  which  have  been  unearthed. 

To  introduce  the  general  concepts  of  a  cost-effectiveness  analysis,  we 
shall  conduct  the  analysis  in  the  simplest  of  terms  --  namely,  we  shall 
determine  how  much  it  costs  to  achieve  a  fixed  effectiveness  for  a  set  of 
two  alternative  system  configurations.  Cost  is  used  to  represent  the  amount 
of  resource  expenditure,  and  effectiveness  is  a  measure  of  the  system's 
ability  to  accomplish  its  mission  objectives. 

The  general  approach  for  making  such  decisions  follows  a  broad  outline 
which  is  summarized  by  the  four  steps: 

•  define  criteria  for  selection, 

•  generate  alternatives  that  satisfy  operational  requirements 
and  constraints, 

•  compute  resultant  values  of  cost  and  effectiveness  for 
each  alternative,  and 

•  evaluate  results  with  respect  to  the  decision  criterion. 


— ^  Sec  Volume  III  of  Task  Group  II,  and  Volume  III  of  Task  Group  IV. 
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See  Volume  III  of  Task  Group  II,  and  Volume  III  of  Task  Group  IV. 


Each  of  these  major  steps  is  discussed  in  some  detail  here.  It  is  worth¬ 
while,  however,  to  set  the  stage  for  such  discussions  in  this  introduction. 

The  criterion  for  selection  must  be  one  that  is  mission  responsive;  that 
is,  it  must  answer  the  right  question.  Essentially,  the  criterion  is  based  on 
maximizing  effectiveness  for  a  given  cost  or,  conversely,  minimizing  cost 
for  a  given  level  of  effectiveness.  However,  it  is  also  necessary  to  define 
the  scope  of  the  analysis  in  terms  of  resources,  system,  operational  and 
support  constraints.  Thus,  the  fundamental  criteria  given  above  naturally 
evolve  into  a  constrained  FOM  such  as,  "maximize  effectiveness  per  dollar, 
provided  effectiveness  is  greater  than  E#  and  cost  is  less  than  C*  (where  E* 
and  C*  refer  to  specific  limiting  values)." 

In  generating  acceptable  alternatives,  identification  of  all  variable  and 
fixed  factors  and  their  costs  is  required.  In  addition,  the  elements  of  risk 
and  uncertainty  as  related  to  these  factors  and  costs  and  the  analysis  of 
effects  on  other  programs  must  also  be  considered.  Such  factors  as  avail¬ 
ability  of  appropriate  data,  computational  capacity,  and  restraints  in  time 
and  effort  available  for  the  analysis  will  play  important  roles  in  this  phase. 

A  generated  alternative  is  then  an  acceptable  combination  of  the  selected 
factors  with  associated  risk  and  uncertainty  elements. 

* 

Measures  of  cost  and  effectiveness  for  each  design  alternative  must 
then  be  computed.  The  form  these  measures  take  is  related  to  the  decision 
criterion.  For  effectiveness,  the  measure  can  range  from  a  simple  prob¬ 
ability  numeric,  to  an  expected  value,  to  the  complete  distribution  of  some 
over -all  performance  characteristic.  The  effectiveness  model  is  based  on 
sub-models  for  reliability,  maintainability,  and  performance.  These  in  turn 
are  based  on  the  variable  and  fixed  factors  to  be  considered  such  as  failure 
and  repair  distributions,  internal  stresses,  environment,  and  design 
integration.  .1- 

The  cost  measure  must  be  one  that  can  treat  the  major  types  of  resource 
expenditures  on  some  common  basis.  Sub-models  are  required  for  develop¬ 
ment  costs,  operating  costs,  and  support  costs  both  in  terms  of  dollars  and 
schedules.  In  addition,  the  burden  that  a  particular  alternative  places  on 


other  systems  and  objectives  must  be  evaluated  for  a  complete  cost  model. 

The  integration  of  the  separate  cost  and  effectiveness  models  into  a 
single  cost-effectiveness  model  provides  the  basis  for  decisions.  It  is  at 
this  stage  that  optimization  theory  becomes  applicable,  involving  such  dis¬ 
ciplines  as  mathematical  programming,  stochastic  process  theory,  calculus 
of  variations,  econometrics,  and  decision  theory. 

All  of  the  above  models  must  satisfy  characteristics  related  to  adequacy, 

'  % 

representativeness,  consistency,  sensitivity,  plausibility,  criticality,  work¬ 
ability,  and  suitability.  In  applying  the  model,  it  must  be  emphasized  that, 
results  of  the  optimization  process  can  only  indicate  the  best  decision  within 
the  simplifications,  assumptions,  restrictions  and  omissions  that  were-  re- 

I 

quired  to  circumvent  such  problems  as  uncertainties,  non -quantifiable 
factors,  and  inadequate  data,  time  or  computational  capacity. 

In  spite  of  these  potential  limitations  upon  the  absolute  accuracy  of  a 
cost-effective  analysis,  the  framework  for  a  final  decision  will  have  been 
provided.  The  cost-effectiveness  analysis  will  have  reduced  the  guess  work 
and  intuitive  estimates  of  cost  and  effectiveness,  and  although  the  initial 
results  must  still  be  critically  evaluated  and  combined  with  relevant  political 
and  timing  factors  by  the  decision  maker,  there  will  have  been  a  significant 
step  forward. 

The  example  chosen  for  illustration  is  assumed  to  be  a  preliminary 
(first  cut)  study  conducted  during  the  Conceptual  Phase  in  response  to  a 
recognized  need  stated  in  a  ROC  or  QOR. 

It  should  be  carefully  noted  that  this  particular  example  is  not  intended 
to  serve  as  a  general  model  of  all  the  required  analysis  in  the  Conceptual 
Phase.  The  example  is  presented  step  by  step  as  a  paraphrase  of  the 
fifteen  principal  tasks  of  the  activity  network  of  Figure  14,  page  125;' 

1 .  0  MISSION  DEFINITION 

1.  1  Functional  Description  A  required  operational  capability  (ROC) 
has  been  identified  in  the  area  of  a  small,  mobile  weapons  launcher  for  use 
at  target  ranges  of  50  to  500  miles  against  soft  targets  of  small  area. 
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mobility  is  required  since  the  anticipated  use  of  the  weapon  will  occur  in  the 
forward  areas  of  a  moving  battle  front. 


1.  2  System  Quantitative  Requirements  The  purpose  of  the  preliminary 
study  is  to  determine  whether  to: 

(1)  proceed  with  a  new  SOR, 

,  (2)  respond  to  the  ROC  by  modifying  an  existing  system,  or 

(3)  postpone  development  in  favor  of  additional  study  and/or 
exploratory  development. 

At  this  stage  there  are  no  firm  system  quantitative  requirements.  Ten¬ 
tative  quantitative  requirements  will  be  the  desired  output  of  the  study  if 
Number  1  is  elected. 

2.  0  IDENTIFY  RESOURCES 

Ordinarily  resources  would  not  be  specified  in  a  first  cut  analysis  in  the 
Conceptual  Phase.  Clearly,  however,  by  the  end  of  the  Conceptual  Phase,  a 
System  Program  Office  must  be  manned  and  a  program  funded.  Potential 
budget  and  state-of-the-art  limitations  are  factors  to  be  reckoned  with. 

3.0  SYSTEM  DESCRIPTION 

At  this  stage  of  analysis  a  number  of  possible  system  configurations  are 
ordinarily  considered,  depending  upon  the  ingenuity  brought  to  bear  upon 
satisfying  the  ROC.  The  descriptions  will,  in  general,  be  very  gross 
(tentative)  and  quite  diverse  in  character.  In  the  following  we  shall  show 
how  two  possible  candidates  among  many  are  compared  as  part  of  the  weed¬ 
ing  out  process  (exhaustion  of  alternatives)  performed  during  the  Conceptual 
and  early  Definition  Phases. 

3.  1  System  Number  1  One  weapon  unit  of  this  proposed  system  will 
consist  of  N  launchers,  each  of  which  is  to  contain  K  missiles. 

The  concept  of  this  system  calls  for  the  assignment  of  one  launcher  to 
a  target  with  up  to  (N-l)  launchers  held  in  reserve.  The  missile(s)  which 
are  stored  in  the  launcher  may  be  assumed  to  be  available  (nonfailed)  if  the 
launcher  is  available. 
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N  and  K  will  be  specified  by  the  analysis  about  to  be  conducted.  The 
launrher  is  mobile,  and  it  is  believed  that  any  of  several  existing  GEM  or 
VTOL  vehicles  could  be  modified  for  use.  The  proven  availability  of  these 
vehicles  under  simulated  tactical  conditions  is  known  to  be, 

a  >  0.  67 

The  proposed  missile  would  be  a  short  range,  solid  propellant  type 
similar  to  several  existing  missiles  whose  proven  dependability  x  capability 
under  simulated  tactical  conditions  is  known  to  be, 

R  >  0.  60 

One  missile  of  this  type  is  capable  of  destroying  the  class  of  target 
considered.  The  definition  of  effectiveness  for  this  system  is: 

E^  =  the  probability  of  target  destruction  when  an  execution 
directive  is  received  at  a  random  point  in  time,  and  one 
launcher  out  of  N  is  selected  for  use. 

3.  2  System  Number  2  There  is  an  existing  IRBM  which,  though  it  is 
normally  a  fixed  emplacement  missile,  could  be  altered  by  modifications  to 
become  portable  by  barge. 

Therefore,  although  this  system  would  not  be  as  mobile  as  system 
Number  1,  its  additional  range  capability  would  offset  the  greater  mobility 
of  the  proposed  system. 

The  IRBM  under  consideration  has  a  proven  effectiveness  under  simu- 
lated  tactical  conditions  of  approximately  .75  as  a  fixed  emplacement 
weapon.  It's  effectiveness  as  a  portable  weapon  will  be  somewhat  less,  but 
the  amount  less  is  expected  to  be  trivial. 

The  definition  of  effectiveness  (E^)  for  the  IRBM  is: 

=  probability  of  target  destruction  when  an  execution 
u  directive  is  received  at  a  random  point  in  time  and 

one  missile  is  assigned  to  a  target. 
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SPECIFY  FIGURES  OF  MERIT 


For  this  first  cut  analysis  the  figure  of  merit  assumes  the  form  of  a 
general  injunction  to  "predict  the  life  costs  of  system  Number  1  (which  can 
satisfy  the  ROC)  and  compare  them  to  the  cost  of  modifying  the  existing 
IRBM  system  to  accomplish  the  same  mission." 

Thus,  a  conservative  figure  of  merit  for  this  preliminary  study 
becomes,  "minimize  the  life  cost  of  a  mobile  launcher -missile  system 
subject  to  the  constraint  E  ^  >  0.75." 

This  figure  of  merit  will  permit  the  life  costs  of  the  proposed  system 
to  be  compared  to  the  costs  of  support  and  modification  of  the  existing  fixed 
emplacement  IRBM  system. 

If  the  costs  are  significantly  different,  a  clear  choice  exists.  If  the 
costs  are  similar,  further  system  definition  and  analysis  will  be  required. 

If  a  clear  choice  exists,  the  present  analysis  would  be  succeeded  by 
another  comparing  the  "winner"  to  another  contender.  This  comparison  by 
pairs  would  continue  throughout  the  Conceptual  and  Definition  Phases  until 
all  proposed  alternatives  have  been  eliminated. 

5.  0  SPECIFY  ACCOUNTABLE  FACTORS 

In  a  first  cut  analysis  of  the  type  being  presented  here,  only  the  highest 
level  factors  are  usually  accounted  for.  We  shall  explicitly  include  the 
following  in  this  analysis: 


'■stem  Number  1 


a  -  availability  of  an  individual  launcher 

A  -  availability  of  one  weapon  unit 

Cj  -  cost  of  one  corrective  maintenance  action  excluding 
fixed  costs 


Ca  -  estimated  cost  of  the  availability  "a"  of  one  launcher 


-  estimated  cost  of  obtaining  a  weapon  unit  availability  of  "A" 


Cj  -  total  weapon  system  investment  cost 


-  fixed  costs  of  weapon  maintenance  and  support 
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u 


R 


u 


-  estimated  cost  of  obtaining  a  net  weapon  unit  dependability 
and' capability  of  "p" 

-  nonrecurring  cost  of  producing  one  weapon  unit  having  a  unit 

effectiveness  P 
u 

-  total  cost  of  corrective  maintenance 

-  total  weapon  system  support  cost 

-  estimated  cost  of  one  missile  with  net  dependability  and 
capability  "R" 

-  total  weapon  system  cost 

-  incremental  cost  of  supporting  one  weapon  unit,  excluding 
fixed  costs 


K 

M 

N 

P 

P 

u 

R 

*d 

T 


number  of  missiles  per  launcher 
number  of  weapon  units  required 

number  of  launchers  assigned  per  target  1 

net  dependability  and  capability  of  one  weapon  unit,  given 
that  at  least  one  launcher  is  available 

per  unit  system  effectiveness 

product  of  dependability  and  capability  for  a  single  missile 

expected  downtime  of  a  single  launcher  for  a  single  corrective 
maintenance  action 

<3 

total  expected  useful  operational  life  of  system. 


System  Number  2 


C’ 


c' 

s 


estimated  cost  of  obtaining  a  weapon  unit  availability  of  "A" 

total  system  investment  cost 

fixed  cost  of  weapon  maintenance  and  support 

estimated  cost  of  obtaining  a  net  weapon  unit  dependability 
and  capability  "p" 

expected  average  yearly  cost  of  corrective  maintenance  per 
missile 

total  system  support  cost 

total  cost  of  one  modified  IRBM  system 
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C  '  -  incremental  cost  of  supporting  one  IRBM  and  its  barge, 

u  excluding  fixed  costs 

M  -  number  of  required  weapon  units 

T  -  total  expected  useful  operational  life  of  system. 

It  will  be  noted  that  the  number  of  parameters  differ  between  systems 
number  1  and  2.  This  is  to  be  expected  since  less  detailed  estimation 
is  required  for  the  existing  IRBM  than  for  the  nonexistent  GEM  launcher. 

6.0  IDENTIFY  DATA  SOURCES 

(See  8. 0  Acquire  Data) 

7.  0  MODEL  CONSTRUCTION 

7.1  Effectiveness  Equations  for  System  Number  1  Since  the  missile(s) 
which  are  stored  in  the  launcher  may  be  assumed  to  be  available  (nonfailed) 
if  the  launcher  is  available,  the  availability  of  a  weapon  unit  (A)  (for  one 
target)  is  given  by, 

A  =  1  -  (1  -  a)N  (1) 

where: 

a  is  the  availability  of  an  individual  launcher 
(N  -  1)  is  the  number  of  launchers  held  in  reserve  on  the  target.  * 

Although  a  launcher  may  contain  several  missiles,  any  one  of  the 
missiles  is  capable  of  destroying  the  class  of  targets  considered.  There¬ 
fore,  if  R  is  the  product  of  dependability  and  capability  for  a  single  missile 
if  there  are  K  missiles  per  launcher ,  then  the  net  dependability  and 
capability  (p)  of  one  weapon  unit,  given  that  at  least  one  launcher  is 
available  is, 

p  -  1  -  (1  -  R)K  (2) 
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For  this  study  it  is  desired  that  N  and  K  be  selected  in  an  optimum 
manner  (least  cost)  under  the  constraint  that  the  unit  effectiveness  (P  )  is 
such  that,  k 

Pu  =  PA  >  E0  (3) 

where  Eq  is  the  unit  effectiveness  of  the  existing  fixed  emplacement  IRBM 
missile  system  which  could,  with  some  modification  be  used  against  the 
class  of  targets  considered. 

E  =  0.  75  (4) 

o 


C  =  M  C  +C  +  C  (5c) 

s  u  r  o 

where 

C  =  total  cost  of  corrective  maintenance 
r 

M  =  number  of  weapon  units  required 

Cp  =  nonrecurring  cost  of  producing  one  weapon  unit  having  a  per 
u  unit  effectiveness  P 

u 

G  -  incremental  cost  of  supporting  one  weapon  unit  (excluding 
fixed  costs  of  weapon  maintenance/support 

f-  -■  fixed  costs  of  weapon  maintenance/support. 
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Gust  Estimating  Relationship  for  Cp  tor  System  Number  1 


The  cost  Cp  for  the  above  system  may  be  expressed  as, 


V  =  CA+Cp 
u  r 


We  shall  assume  that  to  a  first  order  approximation, 


C.  =  NC  ' 
A  a 


C  =  KC„ 

p  R 


(6) 

(7) 

(8) 


where 

C 


A 


estimated  cost  of  obtaining  a  weapon  unit  availability  A 

C  =  estimated  cost  of  obtaining  a  weapon  unit  dependability  x  cap- 
^  ability  of  p 

Ca  s  estimated  cost  of  obtaining  the  availability  "a"  of  one  launcher 

Cp  =  estimated  cost  of  one  missile  of  dependability  x  capability  of  R. 

Substituting  Equations  (7)  and  (8)  into  (1)  and  (2)  respectively,  we 
obtain,  n 


1 


CR 


T 


(9) 


(10) 


7.  2.  3  Cost  Estimating  Relationship  for  Cr  for  System  Number  1 

The  expected  amount  of  down  time  per  launcher  during  an  expected 
operational  life  time  r"  is  given  by, 

T  (1  -  A)1/N 

Then  the  cost  of  unscheduled  maintenance  C  for  M  N  launchers  is, 

r 

^  MTN  (1  a,1/N 

Cr  =  cf  -tT-  (1  •  A) 
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where 


c.^,  =  cost  of  one  corrective  maintenance  action,  excluding  fixed 

1  costs 

T  -  expected  life  of  the  system 

A  =  availability  of  a  weapon  unit  (excludes  preventive  maintenance) 

t^  ~  expected  downtime  of  a  single  launcher  arising  from  correc¬ 
tive  maintenance. 


7.  3  Cost  of  a  Modified  IRBM  System  (System  Number  2)  The  total 
cost  C,p  nf  a  modified  IRBM  system  is  assumed  to  be  given  by, 


CT  = 

ci +  c; 

(ID 

Ci  = 

M(C;  4  C-) 

(12) 

C1  * 

M  C'  +  C1  +  c'  TM 
u  o  r 

(13) 

where 

Cj  -  investment  cost  =  cost  of  modifying  IRBM,  outfitting  barge 
and  procuring  IRBM  and  barge 

=  support  cost 

C'  =  incremental  cost  of  supporting  one  IRBM  and  barge  (excluding 
u  fixed  costs) 


=  fixed  costs  of  weapon  maintenance /support 

c'  =  expected  average  yearly  cost  of  corrective  maintenance  per 
missile. 

7 . 4  Statement  of  the  Optimization  Problem  for  System  Number  1  It 

is  desired  to  minimize  equation  (5a)  subject  to  the  constraint  (3),  We 

7  / 

shall  do  this  by  means  of  Lagrangian  multipliers,—  In  the  present  instance 
this  calls  for  replacing  the  constraint  (3)  with: 


—  One  of  several  optimization  techniques  recommended  by  the  WSEIAC 


138 


(14a) 


F  =  E  +  S2  O  <  S2  <  1  -  E 
u  o  —  —  o 

2 

where  S  is  a  new  variable.  However,  since  (5a)  increases  monotonically 
with  we  may  replace  (3)  with  the  simpler  constraint 


P 


u 


E 


o 


(14b) 


Given  the  modified  constraint  (14b)  and  the  cost  equation  (5a),  the 
optimization  problem  may  be  stated  as:  Allocate  the  given  unit  effectiveness 
between  A  and  p  in  such  a  way  that  the  sum  of  the  investment  cost  Cj 
and  the  total  life  support  cost  is  a  minimum.  It  should  be  carefully 

noted  that  this  optimization  implies  that  development  dollars  will  be  balanced 
against  support  dollars,  a  course  of  action  which  implies  an  apportionment 
between  Air  Force  commands  made  prior  to  the  Acquisition  Phase  at  a  time 
when  the  effectiveness  of  the  proposed  system  is  at  best  a  projection  based 
upon  historical  generic  data.  Such  an  apportionment  will  be  subject  to  a 
large  amount  of  uncertainty,  of  which  due  note  will  be  made  later. 

7.5  Solution  of  the  Optimization  Problem  for  System  Number  1  The 
solution  of  the  problem  as  stated  above  is  accomplished  as  follows:  Form 
a  new  equation  which  is  the  sum  of  (5a)  and  the  constraint  (14) 

m  =  Cj  +  Cs  +  X(Ap  -  E0)  (15) 


Then  a  necessary  condition  for  an  optimum  solution  to  exist  is, 

0 


3f 

3p 


Hence, 


Sf 

35 

5f 

35 


Sf 

¥ 


=  0 


=  0 


(16) 


acr 


+  XA  =  0 


(17a) 
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shown  in  Table  VII,  where  a  range  of  uncertainty  has  been  indicated  for  C  , 
C^,  and  c^.  It  is  assumed  that  sufficient  historical  data  exists  that  the 
estimates  a,  R,  t^,  Cq,  C^,  E^,  etc.  can  be  accurately  made.  It  should 
be  carefully  noted  that  these  are  highly  questionable  assumptions. 

The  WSEIAC  has  identified  current  data  systems  as  a  primary  problem 
area  in  implementing  a  system  effectiveness  or  cost-effectiveness  program. 
In  particular,  it  is  noted  that  no  adequate  historical  data  sources  currently 
exist.  Consequently  this  example  analysis  cannot  be  conducted  in  general 
under  current  circumstances.  To  alleviate  this  situation  the  WSEIAC 
recommends  the  creation  of  a  System  Effectiveness  Information  Central 
(SEIC)  which  would  preserve  appropriate  historical  data.  Such  data  would 
be  obtained  from  System  Information  Banks  (SIB's)  established  for  eacft 
Air  Force  system  at  the  end  of  the  Conceptual  Phase. 

9.  0  PROCESS  DATA 

< , 

First  cut  studies  of  the  type  being  illustrated  obviously  require  no 
mechanical  data  processing  aids.  During  the  later  program  phases,  how¬ 
ever,  enormous  quantities  of  data  are  apt  to  be  generated.  Electronic  data 
processing  is  a  necessity. 

*> 

10  0  SPECIFY  SCHEDULE 

Schedule  enters  into  Conceptual  Phase  studies  in  a  very  simple  way  -- 
it  can  rule  out  those  alternatives  which  cannot  be  delivered  in  the  near  future. 
In  the  present  analysis,  for  example,  an  estimation  of  the  expected  acquisi¬ 
tion  times  of  system  1  and  2  would  be  made.  It  is  conceivable  thai  the 
lesser  cost-effective  system  might  be  acquired  on  a  stop-gap  basis  if  the 
acquisition  time  of  the  alternative  system  were  sufficiently  disparate  and  no 
other  alternative  presented  itself.  Thus,  schedule  in  the  Conceptual  Phase 
is  a  "GO,  NO-GO"  type  of  constraint  which  generally  would  not  appear  expli¬ 
citly  in  cost-effectiveness  equations. 

11 . 0  MODEL  EXERCISE 

H.l  System  Optimization  for  System  Number  1  The  optimum 
choice  for  p  is  found  by  solving  Equation  (19)  using  the  estimates  of  Table  VII. 
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TABLE  VII 


PARAMETER  ESTIMATES  FOR  EXAMPLE 


Proposed  New  System 


Modified  IRBM 


a 

> 

0.  67 

E 

u 

< 

0.  75 

R 

> 

0.  60 

CA 

= 

15  x  106 

ld 

- 

1  day 

t 

C 

P 

= 

20.  4  x  106 

5. 92  x  106 

> 

C  >  4.  58  x  1 06 
a  — 

1 

c 

u 

s 

0.  15  x  106 

$7.  1  x  106 

> 

CR  >  $5.  4  x  106 

Co 

= 

3  x  106 

Co 

= 

$1.  5  x  109 

C' 

r 

s 

4.  4  x  106 

C 

u 

= 

$0.  1  x  106 

$1. 096  x  10 

3 

<  c£  <  $2.  192  x  103 

T  =  10  years 

(given) 

T  = 

10 

years  (given) 

M  =  100 

(given) 

M  = 

100  (given) 

The  eight  possible  optimum  solutions  for  p  indicated  by  the  range  of 
uncertainty  of  the  data  of  Table  VII  are  shown  in  Table  VIII. 

Because  of  the  uncertainty  in  the  data,  the  optimum  choice  (Bee 
Table  VIII)  of  p,  A,  N,  K,  and  costs  for  100  weapon  units  cannot  be  defined 
any  closer  than, 


0.845  < 

P 

< 

0.882 

0.888  > 

A 

> 

0.850 

1. 97  > 

N 

> 

1.71 

2.  03  < 

K 

< 

2.  33 

$2. 007  x  109  < 

CI 

< 

$2,631  x 

109 

$1. 744  x  199  < 

Cs 

< 

$2. 007  x 

109 

$3. 762  x  109  < 

CT 

< 

$4,619  x 

109 

11.2  Cost  of  SyBtem  Number  2  From  Equations  ( 1 1),  (12)  and  (13), 
and  Table  VII  we  obtain  for  100  targets, 

Cj  =  $3.54xl09 

C'  =  $4,418  x  109 

s 

C^.  =  $7,958  xlO9 


11.3  Costs  as  a  Function  of  Number  of  Targets  M  Although  the 
number  of  targets  M  has  been  specified,  this  requirement  is  also  subject 
to  uncertainty.  The  relative  costs  of  system  number  1  and  2  as  a  func¬ 
tion  of  the  number  of  targets  is  shown  in  Figure  15. 

11.4  Interpretation  of  Results  In  spite  of  the  uncertainty  in  the  data 
for  system  number  1,  this  system  should  cost  about  one  half  the  amount  of 
system  number  2  for  coverage  of  100  targets.  However,  it  should  be  noted 
from  Figure  15  that  both  systems  are  equally  cost-effective  for  about  30 
targets,  and  that  system  number  2  is  most  cost-effective  below  27  targets. 
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TABLE  Vm 
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1  I  I  1  t 

50  60  70  80  90 

M 

S  15 


SYSTEM  NUMBER  1  AND  2 
NUMBER  OF  TARGETS 


BLOCK  12.0  MANAGEMENT  SUMMARY  REPORTS 


The  purpose  of  a  management  summary  report  is  to  apprise  the  decision 
maker  of  all  facts  and  conjectures  relevant  to  the  potential  solution  of  a 
problem. 

In  the  present  example,  the  problem  is  to  satisfy  the  ROC  or  QOR  by 
either  selecting  between  potential  system  configurations  or  by  entering 
exploratory  development.  Thus,  the  content  of  a  summary  report  of  the 
present  example  analysis  would  contain  at  least  as  much  as  has  been  pre¬ 
sented  here  for  three  reasons: 

•  a  well  informed  management  is  more  apt  to  make  decisions 
that  withstand  the  cold  clarity  of  hindsight 

> 

•  a  review  of  previous  analyses  becomes  necessary  if  the 
uncertainty  aspects  of  a  ROC  or  QOR  change 

•  the  Conceptual  Phase  studies  should  be  available  for  guidance 
during  later  phases  if  it  is  elected  to  proceed  with  a  TSOR. 

The  recommended  content  of  a  documentation  of  a  Conceptual  Phase 
study  is  that  used  in  this  example;  namely,  a  paraphrase  of  the  first  eleven 
steps  of  the  activity  network  of  Figure  14.  * 


BLOCK  13.0  THE  DECISION  PROCESS 


The  rational  process  of  cost-effectiveness  selection  between  competing 

alternatives  must  eventually  terminate  with  the  exhaustion  of  alternatives. 

In  many  real  situations  the  proper  decision  will  not  be  as  clear  cut  as  we 

have  (conveniently!)  chosen  to  illustrate  it.  All  the  available  facts,  including 

the  estimates  of  uncertainty,  have  been  evaluated,  and  we  choose  system 

number  1.  Notice  that  even  here,  however,  we  have  had  to  exercise  that 

8  / 

indispensable  prerogative  of  management  -  judgment.— 

Before  we  pat  ourselves  on  the  back,  howeve. ,  let  us  pursue  this  deci¬ 
sion  business  a  bit  further.  Now  that  the  bathing  beauty  contest  is  over, 
let's  examine  the  measurements  of  the  winner.  Specifically,  what  set  of 
quantitative  requirements  do  we  put  in  the  TSOR?  The  achievable  value 
Eu  =  .7&  which  was  the  basis  of  our  comparative  analysis?  Something 
less?  Something  better?  How  do  we  decide  what  probability  values  are 
minimum  acceptable? 

How  much  is  good  enough? 

WSEIAC  does  not  tell  us.  They  identify  this  as  a  problem  area  and 
recommend  that  the  problem  be  studied. 

How  serious  a  problem  is  it?  Can  it  be  ignored?  We  may  judge  from 
the  following  WSEIAC  observation. 

,  The  minimum  acceptable  quantitative  requirements  of  a  certain  recent*' 
SOR  are  given  piecemeal  in  terms  of  separate  probabilities  and  perform¬ 
ance  limits  without  obvious  relation  one  to  another.  When  combined  in  an 
over-all  system  effectiveness  number  (along  WSEIAC  lines)  these  require¬ 
ments  suggest  that  if  this  system  works  less  than  4  times  out  of  100,  it  is 
acceptable!  Would  you  cross  a  street  if  those  wore  your  odds  in  traffic? 


8/ 


In  our  judgment  there  will  never  be  less  than  32  targets. 
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14,0  IMPLEMENT  DECISION 

In  view  of  the  comments  under  13,0  The  Decision  Process,  and 
assuming  that  all  candidate  systems  have  been  evaluated,  the  next  course  of 
action  is  as  follows: 

•  establish  a  TSOR  for  a  mobile  missile  launcher  of  the  GEM 

type  with  a  unit  effectiveness  >  0.75,  (The  intent  is  to 

firm  up  during  the  Definition  Phase. ) 

•  establish  an  SIB  for  the  new  system 

•  document  the  Conceptual  Phase  studies  and  include  them  in 
the  TSOR  by  reference 

•  budget  resources 

•  establish  a  schedule  for  initial  operational  capability, 

15.0  CHANGE  ANALYSIS 

(not  pertinent) 
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APPENDIX  III 


GLOSSARY  OF  SYSTEM  EFFECTIVENESS 
AND  COST-EFFECTIVENESS  TERMS 

This  glossary  of  terms  contains  definitions  of  those  words  which  are 
commonly  used  in  the  WSEIAC  Task  Group  reports,  but  which  may  be 
unfamiliar  to  the  reader  or  not  of  standardized  usage.  Those  words  whose 
moaning  is  evident  from  the  context  or  which  hav.e  come  into  standard  usage 
among  the  various  technical  disciplines  are  not  defined  here. 


Accountable  factors  Those  physical  and  organizational  facts  pertaining  to 
an  item  and  its  operational  environment  which  are  specifically  considered 
in  the  construction  of  a  model.  For  example;  failure  rate,  repair  time, 
manning,  maintenance  policy,  boundary  conditions  and  constraints. 

Acquisition  Phase  The  third  of  the  four  phases  of  a  system  life -cycle.  It 
starts  after  issuance  of  the  System  Program  Directive  and  ends  with 
acceptance  by  the  user  of  the  last  operating  unit  in  a  certain  series,  or 
until  the  Specific  Operational  Requirement  has  been  demonstrated  through 
Category  n  testing  and  all  required  updating  changes  resulting  from  the 
testing  have  been  identified,  approved,  and  placed  on  procurement,  which¬ 
ever  occurs  later.  It  subsumes  system  development  and  production. 

Alert  That  portion  of  uptime  when  an  item  is  in  a  vigilant  state  and  is 
thought  to  be  non-failed  and/or  is  waiting  the  execution  directive  to  per¬ 
form  its  intended  mission. 

Apportionment  To  divide  and  assign  an  index  or  portion  of  the  whole  among 
its  constituent  parts  or  elements. 

Availability  A  measure  of  the  system  condition  at  the  start  of  the  mission, 
when  the  mission  is  called  for  at  an  unknown  (random)  point  in,  time. 

•  True  pointwise  availability  -  The  probability  that  a  system  is,  in  fact, 
usable  at  a  specific  point  in  time.  (It  should  be  carefully  noted  that  for 
a  system  to  be  truly  available,  it  must  not  only  be  thought  to  be  usable 
but  must,  in  fact,  be  in  a  usable  condition). 

•  Apparent  pointwise  availability  -  The  probability  that  a  system  is 
apparently  usable,  but  may,  in  fact,  be  non-usable. 

•  Interval  availability  -  Either  true  or  apparent,  is  the  average  of  true  or 
apparent  pointwise  availability,  respectively,  over  a  specified  interval  of 
time  T. 

•  Steady  state  availability  -  The  limiting  value  of  interval  availability  as 
the  time  interval  T  increases  without  bound. 


Mathematically,  these  definitions  are  expressed  as  follows: 

For  the  simple  case  where  a  system  is  down  when  failed,  and  up 
when  non -failed  (e.g.,  no  preventive  maintenance  or  other  type  interup- 
tions),  and  X  is  failure  rate,  p.  is  repair  rate  {exponential  repair  and 
failure  distributions): 


up 


a 


down 


Let  P  (t)  =  pointwisc  availability. 

Then  Pd(t)  =  1  -  P  (t)  =  probability  of  being  in  repair. 


Now  if 

P  (0)  =  probability  of  being  up  at  time  zero,  and 

F^fO)  =  probability  of  being  down  at  time  zero  where 

P  ,  (t)  =  the  conditional  probability  of  being  up  at  time  t; 
u/u  given  you  are  initially  up,  and 

P  ,  ,(t)  =  the  conditional  probability  of  being  up  at  time  t; 
u'  given  you  are  initially  down,  then 

V*>  ■  Pu|0)  pu/u<‘>  +  pdl°>  +  pu/d<*>- 


Thus  it  can  be  shown  that 

Pu/d(t)  =  s”r-x  ‘  ITT-* 

pu/u(t)  pi  +  X  *  n  +  X 
Interval  availability  Aj  is  then, 

f  T 

AI  '  T  J0  Pu<*>  dt 


e- (\  +  P)t 

-  (|J,  +  X)t 
“  • 
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and  steady  state  availability  AL®]  is  given  by, 

•'  A[®J  =  lim  A_  =  lim  P  (t) 

I  u' 

T  -*  00  t  ■*  00 

if 

if  lim  P^(t)  exists.  The  limit  will  usually  exist  unless  strictly  periodic 
preventive  maintenance  occurs  on  the  system..  In  this  case,  however, 

A^  still  will  exist  and  therefore  AL®]  exists. 

An  estimate  AL®]  of  AL®J  may  be  obtained  from  the  following 

relationship:  MTBSI. 

Al>]  a  TCT 

MTBSI  =  mean  time  between  system  interuptions 
TCT  =  total  calendar  time  of  observation. 

* 

Calendar  time  The  total  number  of  calendar  days  or  hours  in  a  designated 
period  of  observation. 

Capability  A  measure  of  the  ability  of  a  system  to  achieve  the  mission 
objectives,  given  the  system  condition(s)  during  the  mission.  It  specifi¬ 
cally  accounts  for  the  performance  spectrum  of  a  system. 

Conceptual  Phase  The  first  of  the  four  phases  of  a  system  life -cycle.  It 
is  initiated  by  a  statement  of  a  general  need  for  a  particular  operational 
capability  in  an  ROC  or  QOR.  The  phase  extends  from  determination  of  a 
broad  objective  or  need,  to  Air  Force  approval  of  the  Program  Change 
Proposal  covering  the  second  phase  of  the  system  life-cycle. 

Constraint  A  bound  or  restraint  on  a  parameter,  variable,  factor,  function 
or  operation. 

Corrective  maintenance  That  maintenance  performed  to  restore  an  item  to 
a  satisfactory  condition  by  providing  correction  of  a  malfunction. 

Corrective  maintenance  time  The  time  that  begins  with  the  observation  of  a 
malfunction  of  an  item  and  ends  when  the  item  is  restored  to  a  satisfactory 
condition.  It  may  be  subdivided  into  'active  maintenance  time'  and  'delay 
time'  and  does  not  necessarily  imply  equipment  or  system  downtime  if 
alternate  modes  of  operation  or  redundancy  are  used. 

Cost  -effectiveness  A  term  used  to  relate  estimated  or  assessed  cost  to 
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estimated  or  assessed  effectiveness.  It  is  the  value  received 
(effectiveness)  for  the  resources  consumed  or  applied  (cost). 

Cost  estimating  relationship  (CER)  A  functional  expression  that  relates  cost 
to  a  variable  or  set  of  variables;  e.g.,  cost  per  pound  of  jet  engine  thrust. 

Cost  optimization  The  process  of  seeking  a  minimum  cost  program  where¬ 
in  effectiveness  is  ordinarily  an  unconstrained  variable. 

Data  clement  (basic)  A  discrete  measurement  or  item  usually  used  as  a 
block  entry  on  a  r ('porting  form. 

P-Ul.a.1  dement  (computational)  A  computed  output  utilizing  two  or  more 
basic  data  elements. 

Definition  Phase  The  second  of  the  four  phases  of  a  system  life-cycle.  It 
is  initiated  by  a  System  Definition  Directive  and  ends  with  issuance  of  a 
System  Program  Directive.  The  purpose  of  this  phase  is  to  refine  a 
grossly  defined  system  down  to  the  subsystem  level. 

Dependability  A  measure  of  the  system  condition  at  one  or  more  points 
during  the  mission;  given  the  system  condition(s)  at  the  start  of  the 
mission.  It  may  be  stated  as  the  probability  (or  probabilities  or  other 
suitable  mission  oriented  measure)  that  the  system  (1)  will  enter  and/or 
occupy  any  one  of  its  significant  states  during  a  specified  mission  and,  (2) 
will  perform  the  functions  associated  with  those  states. 

Downtime  That  portion  of  calendar  time  during  which  the  itom  is  not  in 
condition  to  perform  its  intended  function. 

Environment  The  aggregate  of  ail  conditions  and  influences  which  affect 
the  operation  of  an  item;  e.g.,  physical  location,  temperature,  humidity, 
pressure,  shock,  etc. 

Failure  The  inability  of  an  item  to  perform  its  intended  function.  (The 
intended  function  must  be  specified.  All  failures  are  assumed  to  have  an 
assignable  cause.) 

Failure,  dependent  (secondary)  A  failure  caused  by  the  malfunctioning  of 
associated  item(s). 
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Failure,  independent  (primary)  A  failure  which  is  unrelated  to  the  mal¬ 
functioning  of  associated  item(s). 

8  / 

Failure,  Mean  Time— 'Between  (MTBF)  The  average  (mean)  time  between 
failures  of  repairable  items  calculated  from  the  total  operating  time  and 
the  total  population. 

Failure,  random  A  failure  which  is  random  with  respect  to  its  time  (or 
cycles,  etc.)  of  occurrence.  It  is  also  used  to  denote  failures  that  arise 
from  an  exponential  failure  distribution. 

Failure  rate  The  number  of  failures  of  an  item  expressed  as  a  relation¬ 
ship  to  a  measure  of  life.  The  failure  rate  of  a  probability  distribution  of 
units -to-failure  is  mathematically  defined  as  the  (conditional)  probability 
density  function  of  units -to-failure,  given  that  the  device  has  not  failed 
prior  to  a  given  unit  "  u."  For  example,  if  f(t)  is  the  (absolute)  proba¬ 
bility  density  function  of  times -to-failure,  and  dt  is  some  small  interval 
of  time  starting  at  t,  the  f(t)  dt  represents  the  proportion  of  a  population  of 
devices  starting  at  time  t  which  fail  in  the  time  interval  (t,  t  +  dt).  If 
F(t)  is  the  cumulateive  distribution  of  times -to -failure,  then  the  failure 
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rate  lambda  (X)  is  expressed  as  X  =  |  3  RfFT 

In  the  case  of  exponentially  distributed  times -to-failure,  the  failure  rate 
X  equals  1/m,  where  m  is  the  mean  time  between  failures.  The 
failure  rate  X  in  any  period  of  time  can  be  computed  by  taking  the  ratio 
of  the  failures  f  during  the  operating  period  to  the  number  of  equipments 
N  at  the  start  of  the  operating,  that  is  X  (t)  =  f/N,  This  figure  of  merit 
is  sometimes  referred  to  as  the  failure  hazard,  instantaneous  failure  rate, 
hazard  rate  or  hazard  function.  For  a  mathematical  treatment  refer  to 
Lloyd,  David  and  Lipow,  Myron,  Reliability:  Management,  Methods,  and 
Mathematics,  Space  Technology  Series;  Prentice  Hall,  1962,  p.  130,  ff. 

Failures  (wearout)  Those  failures  which  occur  as  a  result  of  deterioration 
processes  or  mechanical  wear  and  whose  rate  of  occurrence  increases 


—  The  definition  holds  for  time,  cycles,  miles,  events  and  other  units  of 
life  measurement. 
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with  time.  Wenrout  failures  are  those  failures  that  occur  generally  near 
IW  .-nd  of  life  of  an  item  and  are  usually  characterised  by  chemical  or 
mechanical  changes.  These  arc  failures  which  could  have  been  prevented 
hy  a  replacement  policy  based  on  the  known  wearout  characteristics.  A 
specific  example  would  be  motor  brush  wenrout. 

Figure  of  Merit  (FOM)  A  statement,  either  verbal  or  analytic,  which 
relates  mission  objectives  to  quantitative  system  requirements.  It  is  a 
statement  of  the  ability  of  a  system  to  meet  an  operational  need  which 
includes  the  re.cognition  of  risk  and  uncertainty.  A  specified  value  of 
system  effectiveness  is  an  FOM,  The  injunction  to  minimize  program 
costs  subject  to  a  specified  constraint  on  program  objectives  is  also  an 
FOM. 

Human  factors  Human  psychological  characteristics  relative  to  complex: 
systems,  and  the  development  and  application  of  principles  and  procedures 
for  accomplishing  optimum  man-machine  integration  and  utilization.  The 
term  is  used  in  a  broad  sense  to  cover  all  biomedical  and  psychosocial 
considerations  pertaining  to  man  in  the  system. 

Lethality  The  probability  that  weapon  effects  will  damage  the  military 
objective  to  a  specified  degree. 

Maintainability  Maintainability  is  a  characteristic  of  design  and  installation 
which  is  expressed  as  the  probability  that  an  item  will  conform  to  specified 
conditions  within  a  given  period  of  time  when  maintenance  action  is  per¬ 
formed  in  accordance  with  prescribed  procedures  and  resources.  It  is 
denoted  by  the  symbol  M, 

Mean -time -bet  ween -failure  (MTBF)  The  statistical  mean  of  the  distribution 
of  time  between  successive  failures  excluding  downtime.  The  summation 
of  time  between  failures  excluding  downtime  during  a  given  time  period 
divided  by  the  total  number  of  failures  during  the  same  interval  is  an 
estimate  of  the  MTBF,  (See  'failure') 

Moan-time-betwc.on-BVBtem  -Interruptions  (MTBSI)  The  statistical  mean  of 
the  distribution  of  the  time  between  system  interruptions.  The  total  uptime 
in  a  given  time  period  divided  by  the  total  number  of  system  interruptions 
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in  the  same  time  period  is  an  estimate  of  MTBSI. 

Mcan-time-to-repair  (MTTR)  The  statistical  mean  of  the  distribution  of 
corrective  maintenance  time.  The  summation  of  the  durations  of  correc¬ 
tive  maintenance  time  during  a  given  time  period  divided  by  the  total  num¬ 
ber  of  repair  actions  during  the  same  time  period  is  an  estimate  of  MTTR, 

Mission  duration  The  period  of  time  in  which  an  item  is  performing  a 
specified  mission. 

Mission  profile  A  sequential  and  chronological  description  of  the  mission 
of  an  item. 

Model  Any  device,  technique,  or  process  by  means  of  which  the  specific 
relationships  of  a  set  of  quantifiable  system  parameters  maybe  investigated. 

Operational  factors  Various  factors,  generated  by  the  operational  concept, 
which  affect  the  mission  accomplishment.  Among  these  factors  are  the 
number  of  vehicles,  the  availability  requirements,  and  training  require¬ 
ments. 

Operational  Phase  The  period  in  the  system  life-cycle  which  starts  with 
the  delivery  of  the  first  inventory  unit  or  installation  to  the  using 
command  and  terminates  with  disposition  of  the  system  from  the  inventory. 

Operational  profile  (See  'mission  profile') 

Optimization  The  process  of  searching  for  the  most  favorable  combination 
of  two  or  more  independent  and  conflicting  variables. 

Parameter  A  quantity  employed  in  an  analysis  as  a  symbolic  representation 
of  a  system  attribute. 

Penetrability  The  probability  that  a  weapon  system  will  survivie  a  defense 
environment  and  arrive  at  the  military  objective  intact.  (See  survivability) 

Preventive  maintenance  That  maintenance  performed  to  retain  an  item  in 
satisfactory  operational  condition  by  providing  systematic  inspection, 
detection  and  prevention  of  incipient  failure  such  as  maintenance  to  per¬ 
form  measurements;  care  of  mechanical  woarout  items;  fro;:t  panel 
adjustment,  calibration  and  alignment;  cleaning;  etc. 
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time  the  execution  directive  is  received. 


Ready  time  The.  period  of  time  during  a  mission  that  the  item  is  available 
for  operation,  but  is  not  required.  (Different  from  'alert'  time) 


Redundancy  The  existence  of  more  than  one  means  for  an  item  to  perform 
a  function,  where  all  means  must  fail  before  there  is  an  over-all  failure  of 
the  item. 


Redundancy,  parallel  That  redundancy  in  which  a  function  is  performed  by 
simultaneous  operation  of  two  or  more  items,  anyone  of  which  is  capable 
of  performing  the  function  alone  in  the  event  of  failure  of  one  or  more  of  the 
other  items. 

Redundancy,  standby  That  type  redundancy  in  which  an  alternate  means  of 
performing  the  task  is  available  and  is  either  switched  on  by  a  malfunction 
sensing  device  when  the  primary  item  fails,  for  example,  or  turned  on 
after  failure  of  the  primary  item. 

Reliability  (R)  The  probability  that  an  item  will  perform  a  required  func¬ 
tion,  under  specified  conditions,  without  failure,  for  a  specified  period  of 
time. 

Reliability,  achieved  A  statistical  estimate  of  reliability  based  on  actual 
demonstration  under  specified  conditions.  The  specified  conditions  may 
be  test  conditions  or  operational  conditions,  but  the  conditions  must  be 
clearly  stated,  (See  'inherent  reliability'  and  Operational  reliability') 

Reliability,  inherent  The  theoretical  maximum  reliability  of  a  design 
assuming  no  design  changes,  and  operation  in  an  ideal,  standard  or 
theoretical  environment;  for  example,  a  standard  summer  day. 

Reliability,  minimum  acceptable  A  reliability  below  which  the  item  is 
considered  unacceptable;  also,  a  contractual  requirement  used  as  a  con¬ 
dition  for  acceptance.  Conditions  of  calculation  and  measurement  must 
be  dearly  stated. 
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Reliability,  operational  The  reliability  of  an  item  when  operating  and  being 
maintained  in  a  specified  operational  environment,  usually  by  military 
personnel.  (See  'inherent  reliability,  'f 

V 

Repair  The  corrective  maintenance  process  of  returning  an  item  to  a 
specified  condition  by  either  repairing  in  place;  removing,  repairing  and 
replacing  the  same  item;  or  by  replacing  with  a  like  serviceable  itepn. 

Resources  Resources  are  men,  money,  materiel,  facilities,  time,  docu¬ 
mentation  and  intangibles  such  as  morale,  skills  and  technology.  Mater¬ 
iel  is  supplies,  spares,  consumables,  equipment,  raw  materials,  tools 
and  the  like.  Documentation  is  documented  procedures,  policy,  engineer¬ 
ing  drawings,  methods,  techniques  and  historical  information  on  events 
and  actions.  Such  activities  as  training  are  a  resource,  but  they  are  not 
considered  resources  in  the  strictest  sense.  Training  enhances  the  value 
of  resources;  e.g,,  increases  the  ability  of  men  to  perform,  or  increases 
the  effectiveness  or  efficiency  or  value  of  documentation  (procedures, 
policy,  etc.),  .but  is  not  itself  a  resource. 

Risk  The  calculable  odds  associated  with  either  a  system  significant  event 
or  achievement  of  program  objectives. 

Skill  levels  The  classification  system  used  to  rate  Air  Force  personnel  as 
to  their  relative  abilities  to  perform  their  assigned  jobs. 

Specific  Operational  Requirement  (SOR)  An  Air  Force  document  containing 
the  definition  of  the  qualitative  and  quantitative  system  requirements. 

State  An  identifiable  and  unique  condition  of  a  system  or.  subsystem. 

Subsystem  A  composite  of  equipment,  skills,  and  techniques  which  per¬ 
forms  a  unique  function  (e.g.,  navigation),  but  which  is  not  self-sufficient 
to  perform  the  complete  operational  role. 

Support  cost  The  cost  in  dollars  or  some  other  suitable  measure,  of  those 
resources  expended  in  the  maintenance  of  an  item.  Note  that  all  resources 
(see  'resources')  are  not  necessarily  expendable  (e.g.,  morale,  documen¬ 
tation)  or  may  not  be  expendable  for  a  particular  situation  except  for 
depreciation  or  obsolescence  (e.g.,  facilities,  equipment). 


SupporL  cusl,  indirect  Those  expended  resources,  while  not  expended 
directly  in  support  of  the  maintenance  operation,  contribute  to  the  over-all 
maintenance  „vs»mon  by  supporting  overhead  operations,  administration, 
facility  records  and  statistics,  supervision,  facilities  upkeep,  etc. 

Survivability  The  proi  ability  that  a  system  will  survive  in  an  attack  envi¬ 
ronment.  The  distinction  between  penetrability  and  survivability  (see 
'pern. tr ability1)  is  a  matter  of  degree  only. 

System  A  compos. te  of  equipment,  skills  and  techniques  capable  of  per¬ 
forming  and/or  supporting  an  operational  role.  A  complete  system 
includes  related  facilities,  equipment,  material,  services  and  personnel 
required  for  its  operation  to  the  degree  that  it  can  be  considered  a  self- 
sufficient  unit  in  its  intended  operational  and/or  support  environment. 

System  effectiveness  A  measure  of  the  extent  to  which  a  system  may  be 
expected  to  achieve  a  set  of  specific  mission  requirements  expressed  as  a 
function  of  availability,  dependability  and  capability. 

System  interruption  Any  event  which  removes  the  system  from  an  imme¬ 
diately  usable  condition,  Failures  (known  or  unknown),  planned  or 
unplanned  maintenance,  and  a  variety  of  administrative  actions  such  as 
crew -training  exercises,  are  all  potential  causes  of  system  interruption. 

System  life-cycle  A  system  is  considered  to  evolve  through  four  relatively 
distinct  phases: 

■  conceptual  (feasibility) 

•  program  definition 

•  acquisition 

•  operational. 

System  significant  event  Any  change  of  system  state  which  effects  cost  or 
effectiveness. 


Task  Analysis  An  analytical  process  employed  to  determine  the  specific 
behaviors  required  of  human  components  in  a  man-machine  system.  It 
involves  determining,  on  a  time  base,  the  detailed  performance  required 
of  a  man  and  machine,  the  nature  and  extent  of  their  interactions,  and  the 
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effects  of  environmental  conditions  and  malfunctions.  Within  each  task, 
behavioral  steps  are  isolated  in  terms  of  perceptions,  decisions,  memory- 
storage,  and  motor  outputs  required,  as  well  as  the  errors  which  may  be 

.ir 

expected.  The  data  are  used  to  establish  equipment  design  criteria,  per¬ 
sonnel,  training  requirements,  etc. 

Uncertainty  A  condition,  event,  outcome,  or  circumstance  the  extent, 
value,  or  consequence  of  which  is  not  predictable. 

Uptime  That  portion  of  time  that  an  item  is  available  to  perform  as 
intended. 

Variable  Those  parameters  or  quantities  of  an  analysis  the  use  of  which, 
when  varied,  will  result  in  variations  in  resources,  system  effectiveness? , 
or  the  cost  of  accomplishing  program  objectives. 
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ABSTRACTS  AND  SUMMARIES 
OF 

WSEIAC  REPORTS  BY  TASK  GROUP 

This  section  includes  abstracts  as  they  appear  in  each  of  the  ten  (10) 
volumes  which  comprise  the  final  reports  of  the  five  task  groups.  In 
addition  to  the  abstracts,  brief  summaries  of  the  contents  of  the  reports 
are  presented.  This  will  allow  the  reader  to  obtain  a  grasp  of  the  scope 
and  effort  represented  by  the  entire  study,  which  obviously  cannot  be  pre¬ 
sented  in  detail  in  this  integrated  summary  report. 


A.  TASK  GROUP  I,  "REQUIREMENTS  METHODOLOGY11^  ^ 

1.  Abstract 

The  objective  of  Task  Group  I  was  "To  review  present  proce¬ 
dures  being  used  to  establish  system  effectiveness  requirements  and 
recommend  a  method  for  arriving  at  requirements  that  are  mission  respon¬ 
sive."  Applicable  documents  were  examined,  including  Department  of 
Defense  Directives  and  Instructions,  Air  Force  Regulations,  Manuals, 
Specifications,  Office  Instructions,  etc.,  that  might  be  used  to  establish 
effectiveness  requirements.  Detailed  examination  of  the  Specific  Opera¬ 
tional  Requirement  (SOR)  and  the  companion  Directorate  Office  Instruction 

(DOI)  11-7  resulted  in  the  preparation  of  a  proposed  Air  Force  Manual 

-  1 

(Appendix  I).  This  document  provides  checklists,  guidelines,  and  proce¬ 
dures  for  SOR  preparation  that  include  the  significant  elements  of  system 
effectiveness.  A  proposed  Air  Force  Regulation  (Appendix  II)  was  developed 
to  formalize  a  program  of  effectiveness  evaluation  and  prediction  for  the 
system  life-cycle.  Policy,  concepts,  and  major  command  responsibilities 
are  developed.  Additional  conclusions  and  recommendations  are  submitted 
relative  to  effectiveness  requirements  that  constitute  necessary  steps  to 
development  of  an  Air  Force  wide  system  effectiveness  management 
program. 

2.  Summary 

The  attention  of  Task  Group  I  was  first  directed  at  delineating 
the  scope  of  this  rather  generally  stated  problem.  Use  of  the  term  "require¬ 
ments"  was  interpreted  as  having  reference  to  those  formal  published  Air 
Force  documents  which  are  prepared  during  the  Conceptual  Phase  of  system 
life  and  which  provide  basic  guidance  for  the  more  detailed  management 
documents  prepared  during  the  Definition  and  Acquisition  Phases.  However, 
not  all  of  the  Conceptual  Phase  was  included  in  the  scope  of  the  Task  Group  I 
area  of  interest.  As  officially  defined,  the  Conceptual  Phase  of  development 
includes  the  establishment  of  a  System  Program  Office  (SPO),  the  prepara¬ 
tion  of  a  Preliminary  Technical  Development  Plan  (PTDP)  and  a  Program 
Change  Proposal  (PCP),  terminating  when  the  program  is  approved  by  the 
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Office  of  the  Secretary  of  Defense  (OSD).  Task  Group  I  activity  is  directed 
u.1  only  certain  parts  of  the  first  portion  of  this  Conceptual  Phase.  Feasi¬ 
bility  studies  which  are  often  the  forerunner  of  new  systems,  are  not 
regarded  as  being  a  subject  of  interest.  Advanced  Development  Objectives 
(AI)O's)  are  also  excluded  from  consideration  because  they  arc  by  definition 
directed  at  the  experimental,  not  the  operational  inventory. 

The  documents  which  remain  include  Qualitative  Operational 
Requirements  (QOR's),  Operational  Support  Requirements  (OSR's),  and 
Specific  Operational  Requirements  (SOR's).  The  QOR  is  a  statement  of  need, 
expressed  by  a  command,  and  is  used  as  a  basis  for  preparation  of  an  SOR 
or  OSR.  Doth  of  these  latter  documents  arc  S'ufficiuntly  similar  to  permit 
the  same  general  considerations  of  methodology  and  content  to  be  applied  to 
both.  The  scope  of  the  study  is  therefore  limited  to  the  preparation  of  SOR'b, 
with  the  understanding  that  recommendations  can  be  extrapolated  to  OSR's 
where  appropriate, 

a.  Proposed  Air  Force  Manual  Task  Group  I  reviewed  the 
procedures  proposed  in  the  draft  version  of  DOI  11-7,  AFDORQ,  Adminis¬ 
trative  Practices,  Specific  Operational  Requirement.  This  instruction  did 
not  incorporate  effectiveness  requirements;  consequently,  a  rewrite  of  the 
instruction  was  instituted.  In  the  course  of  this  rewrite,  it  was  apparent 
that  a  systematic  approach  must  be  provided  to  enable  all  SOR  writers  to 
follow  a  standard  procedure,  A  checklist  was  developed  which  embodied 
all  of  the  provisions  of  DOI  11-7  and  added  the  concept  of  weapon  system 
effectiveness.  A  natural  outgrowth  of  the  checklist  was  a  set  of  instructions 
for  its  use,  and  the  two  together  have  evolved  into  a  proposed  Air  Force 
Manual  for  developing  SOR's,  The  proposed  manual  is  included  as  Appendix 
I  to  this  report  and  is  a  principal  output  of  the  task  group.  The  manual  is 
patterned  after  the  present  Department  of  the  Air  Force,  Headquarters 
United  States  Air  Force,  Directorate  of  Operational  Requirements,  Depart¬ 
mental  Operating  Instruction  (DOI  11-7),  The  proposed  manual  subsumes 
all  of  the  administrative  practices  contained  in  DOI  11-7  and  provides: 

(1)  a  comprehensive  checklist  for  writing  an  SOR 

which  covers  the  significant  elements  of  system 
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Office  of  the  Secretary  of  Defense  (OSD).  Task  Group  I  activity  is  directed 
at  only  certain  parts  of  the  first  portion  of  this  Conceptual  Phase.  Feasi¬ 
bility  studies  which  are  often  the  forerunner^of  new  systems,  are  not 
regarded  as  being  a  subject  of  interest.  Advanced  Development  Objectives 
(ADO's)  are  also  excluded  from  consideration  because  they  are  by  definition 
directed  at  the  experimental,  not  the  operational  inventory. 

The  documents  which  remain,  include  Qualitative  Operational 
Requirements  (QOR's),  Operational  Support  Requirements  (OSR's),  and 
Specific  Operational  Requirements  (SOR's).  The  QOR  is  a  statement  of  need, 
expressed  by  a  command,  and  is  used  as  a  basis  for  preparation  of  an  SOR 
or  OSR.  Both  of  these  latter  documents  are^  sufficiently  similar  to  pfernjit 
the  same  general  considerations  of  methodology  and  content  to  be  applied  to 
both.  The  scope  of  the  study  is  therefore  limited  to  the  preparation  of  SOR's, 
with  the  understanding  that  recommendations  can  be  extrapolated  to  OSR's 
where  appropriate. 

a.  Proposed  Air  Force  Manual  Task  Group  I  reviewed  the 
procedures  proposed  in  the  draft  version  of  DOI  11-7,  AFDORQ,  Adminis¬ 
trative  Practices,  Specific  Operational  Requirement.  This  instruction  did 
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not  incorporate  effectiveness  requirements;  consequently,  a  rewrite  of  the 
instruction  was  instituted.  In  the  course  of  this  rewrite,  it  was  apparent 
that  a  systematic  approach  must  be  provided  to  enable  all  SOR  writers  to 
follow  a  standard  procedure.  A  checklist  was  developed  which  embodied 
all  of  the  provisions  of  DOI  11-7  and  added  the  concept  of  weapon  system 
effectiveness.  A  natural  outgrowth  of  the  checklist  was  a  set  of  instructions 
for  its  use,  and  the  two  together  have  evolved  into  a  proposed  Air  Force 
Manual  for  developing  SOR's.  The  proposed  manual  is  included  as  Appendix 
I  to  this  report  and  is  a  principal  output  of  the  task  group.  The  manual  is 
patterned  after  the  present  Department  of  the  Air  Force,  Headquarters 
United  States  Air  Force,  Directorate  of  Operational  Requirements,  Depart¬ 
mental  Operating  Instruction  (DOI  11-7).  The  proposed  manual  subsumes 
all  of  the  administrative  practices  contained  in  DOI  11-7  and  provides: 

(1)  a  comprehensive  checklist  for  writing  an  SOR 

which  covers  the  significant  elements  of  system 
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performance  and  system  effectiveness; 

(?,)  a  format  for  SOR's  defined  so  that  the  sections 
and  the  main  paragraphs  of  every  section  bear 
the  same  number,  title,  and  contents; 

(3)  a  basis  for  quantitative  requirements  to  be  entered 
for  all  system  performance  and  system  effective¬ 
ness  elements. 

b.  Proposed  Air  Force  Regulation  The  introduction  of  the 
concept  of  effectiveness  in  an  SOR  must  be  supported  by  a  program  for 
evaluation  and  prediction  during  the  life  of  the  system.  Accordingly,  Task 
Group  I  has  prepared  a  proposed  regulation  (Appendix  II)  which  calls  for 
such  a  program  and  assigns  command  responsibilities.  During  the  prepara¬ 
tion  of  this  proposed  regulation,  meetings  were  held  with  representatives 
from  Task  Group  II  in  order  that  the  concepts  and  definitions  used  by  that 
Task  Group  would  be  reflected  in  the  draft.  It  should  be  noted  that  this 
regulation  calls  for  both  an  evaluation  program  and  an  assurance  program 
prior  to  evaluation.  In  addition,  it  provides  for  experience  retention  (and 
information  dissemination)  through  establishment  of  a  System  Information 
Dank  (SID)  for  each  new  system,  and  a  System  Effectiveness  Information 
Central  (SEIC)  which  will  be  responsible  for  final  storage  and  retrieval  of 
all  system  effectiveness  information. 


B. 


TASK  GROUP  II,  "PREDICTION  -  MEASUREMENT" 

VOLUME  I  m 

SUMMARY,  CONCLUSIONS  AND  RECOMMENDATIONS'  1 

ir 

1.  Abstract 

Concepts  of  system  effectiveness  measurement  and  prediction, 
presented  in  detail  in  Volume  II,  are  summarized  briefly  in  this  volume. 
Eight  formalized  tasks  necessary  to  evaluate  effectiveness  are  reviewed. 
Summaries  of  four  illustrative  examples,  presented  in  detail  in  Volume  III, 
are  given.  These  examples  provide  useful  guidelines  for  effectiveness 
evaluation  at  various  phases  of  system  life-cycle.  Conclusions  concerning 
the  present  state  of  system  effectiveness  evaluation  are  presented.-  A  series 
of  recommendations  are  proposed  for  Air  Force  adoption. 

VOLUME  II 

CONCEPTS,  TASK  ANALYSIS,  PRINCIPLES 
OF  MODEL  CONSTRUCTION  (3) 

1.  Abstract 

Concepts  of  system  effectiveness  including  the  three  principal 
terms,  availability,  dependability,  and  capability,  are  presented.  I£ight 
specific  tasks  required  to  evaluate  effectiveness  during  any  phase  of  system 
life  are  presented.  A  mathematical  structure  appropriate  to  effectiveness 
model  construction  is  described.  Using  the  above  task  analysis  and  the 
model  framework,  a  hypothetical  example  is  presented.  Results  of  the 
evaluation  illustrate  effectiveness  analysis  methods  and  possible  alternate 
decisions  available.  Application  of  simulation  methods  to  the  example  are 
discussed.  The  appendixes  contain  summaries  of  four  typical  examples  of 
the  application  of  effectiveness  evaluation  methods  to  various  Air  Force 
systems  (presented  in  detail  in  Volume  III).  An  airborne  avionics  system, 
an  intercontinental  ballistic  missile  system,  a  long  range  radar  surveillance 
system,  and  a  spacecraft  system  are  described. 

2.  Concept  of  Effectiveness 

System  effectiveness  concepts  adopted  by  Task  Group  II  are 
summarized  below: 
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v  V 

System  Effectiveness  is  a  measure  of  the  extent  to  which  a 
system  may  be  expected  to  achieve  a  set  of  specific  mission 

,ir 

requirements  and  is  a  function  of  availability,  dependability 
and  capability. 

Availability  is  a  measure  of  the  system  condition  at  the  start 
of  a  mission  and  is  a  function  of  the  relationships  among  hard¬ 
ware,  personnel  and  procedures. 

Dependability  is  a  measure  of  the  system  condition  at  one  or 
more  points  during  the  mission;  given  the  system  condition(s) 
at  the  start  of  the  mission  and  may  be  stated  as  the  probability 
(or  probabilities  or  other  suitable  mission  oriented  measure) 
that  the  system  (1)  will  enter  and/or  occupy  any  one  of  its 
significant  states  during  a  specified  mission  and,  (2)  will 
perform  tile  functions  associated  with  those  states. 

Capability  is  a  measure  of  the  ability  of  a  system  to  achieve 
the  mission  objectives;  given  the  system  condition(s)  during 
the  mission,  and  specifically  accounts  for  the  performance 
spectrum  of  a  system. 

The  objectives  of  system  effectiveness  evaluation  are  to: 

(1)  evaluate  system  designs  and  compare  alternative 
configurations 

(2)  provide  numerical  estimates  for  use  in  defense 
planning 

(3)  provide  management  visibility  at  every  phase  of 
a  system's  life  cycle  of  the  extent  to  which  the 
system  is  expected  to  meet  its  operational 
requirements  (SOR). 

(4)  provide  timely  indication  of  the  necessity  for 
corrective  actions 

(3)  compare  the  effect  of  alternative  corrective  actions. 
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Principal  Tasks 


Eight  formalized  tasks  essential  to  evaluation  of  system  effective- 
ni'KR  were  identified  by  Task  Group  II.  These  are  presented  in  the 
accompanying  diagram. 

DEFINE  MISSION 
Functional  description 
System  requirements 

_ la.sk.  JL- - 


DESCRIBE  SYSTEM 
Block  diagram 
Functional  analysis 
Operating  profile 
Maintenance  profile 

Task  2 
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SPECIFY  FIGURES 
OF  MERIT 

Task  3 


IDENTIFY  ACCOUNTABLE  FACTORS 
Level  of  accountability 
Operational/Maintenance  Factors 
Environment  Data  constraints 

TaBk 


ACQUIRE  DATA 
Data  sources 
Data  elements  * 
Test  method  ~ 
Report  system 

Task  6 

I —  3; 


ESTIMATE  MODEL 
PARAMETERS 
Data  transformation  to 
model  requirements 

ask  7 


CONSTRUCT  MODEL 
Assumptions,  definitions 
Mission  outcomes 
System  states 


Code : 


-(> 


exerci/e  m6del 

Estimate  E 
Comparative  analysis 
Parameter  variation 
Decision  baBis 

Task  8 


- Iterative  process  FIGURE  16 

EIGHT  TASKS  ESSENTIAL  TO  EVALUATION  OF  SYSTEM  EFFECTIVENESS 
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VOLUME  III  ... 

TECHNICAL  SUPPLEMENT1  ' 

1.  Abstract  i 

The  Technical  Supplement  is  concerned  primarily  with  four 
examples  of  effectiveness  evaluations.  The  systems  involved  are:  the 
avionics  system  in  a  tactical  fighter-bomber  (Example  A);  a  squadron  of 
intercontinental  ballistic  missiles  (Example  B);  a  fixed  radar  surveillance 
and  threat  evaluation  system  (Example  C);  and  a  spacecraft  system  (Example 
D).  In  addition  to  the  variety  of  system  types  included,  an  attempt  has  been 
made  to  illustrate  procedures  employed  at  different  program  phases  of  the 
system  life -cycle.  The  evaluation  of  the  avionics  system  takes  place  during 
program  definition;  the  ICBM  squadron,  during  operation;  the  radar  system, 
during  definition  and  operation;  and  the  spacecraft,  during  acquisition. 

Since  evaluation  during  the  Conceptual  Phase  will  generally  be  based  on  a 
gross  comparison  with  existing,  similar  systems,  it  was  not  felt  that  an 
example  of  such  an  analysis  was  necessary.  Further,  each  example  is 
intended  to  illustrate  to  a  different  level  of  detail,  various  aspects  of  the 
evaluation.  The  avionics  system  example,  for  instance,  shows  the  possibility 
of  combining  independent  evaluations  of  several  subsystems.  The  radar 
example  shows  simplifications  which  can  be  made  in  order  to  minimize  the 
number  of  system  states  to  be  considered.  In  the  ICBM  example,  illustra¬ 
tions  of  many  of  the  detailed  procedures  required  to  evaluate  components  of 
the  vectors  and  matrices  are  shown.  Finally,  the  spacecraft  example 
addresses  itself  to  techniques  for  determining  elements  of  the  dependability 
matrix.  It  is  stressed,  however,  that  these  examples  do  not  purport  to 
illustrate  all  possible  methods  of  application  and  use  of  the  evaluation  pro¬ 
cedures.  Rather  they  are  intended  to  show  some  methods  for  applying  the 
concepts,  areas  of  flexibility  in  their  application,  and  some  uses  which 
might  be  made  of  the  evaluations. 

* 

2.  Summary  • 

Short  resumes  of  the  examples  presented  in  Volume  III  of  the 
Task  Group  II  report  follow. 
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Airborne  Avionics  System  Example  The  purpose  of  this  example 
is  to  demonstrate  how  the  effectiveness  evaluation  techniques  proposed  by 
Task  Group  II  may  be  applied  to  the  avionics  system  of  a  tactical  fighter - 
bomber  aircraft.  The  example  considers  only  the  "bombing"  function. 

Similar  analyses  could  be  made  for  its  "fighter,"  "ground  support,"  etc., 
functions. 

It  is  assumed  that  the  effectiveness  evaluation  is  being  made 
during  the  Program  Definition  Phase  of  system  life.  Similar  evaluations 
in  the  real  world  would  also  be  necessary  for  system  configurations  estab¬ 
lished  during  the  Acquisition  and  Operational  Phases.  A  major  considera¬ 
tion  of  the  Program  Definition  Phase  is  "force  structure;"  i.e.,  the'number 
of  systems  (aircraft)  required  to  accomplish  a  specific  mission.  The 
example  illustrates  how  the  results  of  the  effectiveness  evaluation  aid  in 
making  trade-offs  ^nd  ultimate  decisions. 

The  criterion  of  effectiveness  is  as  follows: 

•  At  any  random  time  when  an  execution  order  is 
received,  the  aircraft  shall  take  off  immediately, 
receive  a  target  assignment,  proceed  to  target 
area,  deliver  weapon  within  500  feet  of  target,  and 
return  to  assigned  operation  base. 

ICBM  Fleet  Example  It  is  the  specific  object  of  this  example 
to  illustrate  the  analysis  of  an  ICBM  fleet  in  terms  of  the  formal  mathema¬ 
tical  structure  adopted  by  Task  Group  II  of  the  WSEIAC.  In  particular,  the 
analysis  illustrates  the  usefulness  of  models  in  assessing  the  impact  of 
potential  system  alterations. 

The  criterion  of  effectiveness  for  this  hypothetical  system- may 
be  stated  as  follows: 

•  Any  missile  of  the  ICBM  fleet  should  be  ready  to  accept 
a  launch  directive  at  a  random  point  in  time,  or  at  an 
arbitrary  time  after  an  alarm  condition  has  been  estab¬ 
lished  at  a  random  point  in  time.  It  should  then  launch 
successfully  within  a  prescribed  reaction  time,  fly  a 


ballistic  trajectory,  penetrate  enemy  defenses,  arm, 
fuse,  impact  within  the  prescribed  target  area,  detonate 
and  yield  as  planned  with  a  prescribed  probability  of 
target  destruction. 

Minimum  acceptable  and  objective  numerical  system  require¬ 
ments  for  availability,  countdown,  flight,  and  probability  of  kill  are 
postulated  in  the  form  of  an  SOR.  ' 

Radar  Surveillance  System  Example  This  example  illustrates 
for  this  type  of  defense  system  specific  recommended  effectiveness  predic¬ 
tion  techniques.  The  tasks  required  to  evaluate  system  effectiveness  are 
considered  for  the  four  phases  of  system  life,  and  the  increasing  amount  of 
detail  which  is  necessary  as  the  system  evolves  is  shown. 

The  requirements  of  this  system  are: 

•  detect  airborne  objects  in-the  surveillance  sector  at 
a  range  of  not  less  than  3,000  nautical  miles 

•  identify  the  objects,  and  determine  within  30  minutes 
whether  or  not  they  constitute  a  threat. 

Spacecraft  System  Example  This  example  illustrates  in  some 
detail  a  method  by  which  the  dependability  of  a  spacecraft  may  be  determined 
from  conservative  estimates  of  hardware  reliability.  This  approach  is 
recommended  when  only  small  amounts  of  test  data  on  the  vehicle  are 
available.  This  usually  occurs  during  the  Program  Definition  and  early 
system  Acquisition  Phases  of  programs  on  new  systems.  It  is  also  useful 
for  evaluating  extremely  costly  systems  of  which  only  a  few  are  to  be  con¬ 
structed.  No  effort  is  made  in  this  example  to  treat  availability  or 
capability,  beyond  illustrating  their  tie-in  with  dependability  to  calculate 
effectiveness. 

The  assumed  purpose  of  this  evaluation  is  the  determination  of 
critical  elements  in  the  proposed  spacecraft  configuration.  The  criterion  of 
dependability  for  this  system  is  as  follows: 

■  The  spacecraft  system  shall  be  capable  of  placing  a 


variety  of  payloads,  including  multiple  satellites,  into 
precise  orbits  about  the  earth.  It  shall  have  the  capa¬ 
bility  of  restarting  in  space  after  a  sufficient  coast 

.it 

period  dependent  on  the  specific  payload  and  attitude 
orientation  in  space.  The  system  shall  be  designed 
as  an  upper  stage  rocket  propulsion  vehicle. 


C.  TASK  GROUP. Ill,  "DATA  COLLECTION  AND 

_  MANAGEMENT  REPORTS"1 

1 .  Abstract 

The  report  discusses  the  acquisition,  control,  reporting, 
summarizing  and  retrieving  of  data  as  it  pertains  to  system  effectiveness. 
Section  II  provides  a  summary  of  the  main  body  of  the  report,  which  com¬ 
prises  Sections  III,  IV  and  V.  Section  III  covers  data  acquisition  and  data 
elements  in  detail.  Section  IV  discusses  a  System  Effectiveness  Informa¬ 
tion  Central  (SEIC)  and  a  System  Information  Bank  (SIB)  for  System  Project 
Office  use,  and  in  addition,  presents  some  of  the  problems  on  information 
retrieval.  Section  V  discusses  Management  Summaries,  and  gives  many 
examples  and  suggestions  on  how  to  present  summarized  data.  Numerous 
conclusions  and  recommendations  resulting  from  the  efforts  of  Task  Group 
III  are  incorporated  as  Sections  VI  and  VII.  The  three  appendixes  to  the 
report  contain  respectively,  samples  of  reporting  forms,  a  survey  of 
languages  and  systems  for  information  retrieval,  and  examples  of  proposed 
system  effectiveness  matrixes. 

2.  Summary  of  Section  on  Data  Acquisition 

Initially,  the  groups  identified  and  defined  generated  raw  data 
elements  fundamental  to  the  many  disciplines  (i.e.,  reliability,  maintaina¬ 
bility,  repair  actions,  personnel  subsystems,  etc. )  inherent  in  evaluation 
and  measurement  of  system  effectiveness  and  basic  to  the  mathematical 
models  and  measurement  of  system  effectiveness.  Although  the  data 
elements  identified  are  essentially  those  relating  to  missile  site  or  squadron 
activity,  they  are  easily  translatable  into  comparable  actions  in  support  of 
aircraft  or  electronic  systems/equipment  and  are  considered  minifnum 
requisites  for  developing  system  effectiveness. 

Matrices  have  been  designed  to  display: 

•  raw  data  sources 

•  data  recording  forms 

•  data  collection/reporting  systems  and  controlling  agencies 

•  field -generated  versus  computer -generated  data 

current  data  collection/reporting  systems  coverage. 
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Analysi^-of  these  matrixes  served  to  identify  and  highlight 
major  problem  areas  iriherent  in  the  data  acquisition  systems  presently 
used  in  industry  and  the  Air  Force.  Some  of  the  more  obvious  defeciencies 
are:  » 

(1)  Certain  needed  data  elements  appear  in  all  data  systems, 
but  many  needed  dg.ta  elements  appear  in  one  and  in  no 
other  system. 

(2)  Some  systems  record  alnd  report  essential  data  relating 
to  time  increments  (i.  e.,  AFSC  Form  258  used  mainly 
within  AFSC),  but  most  systems  lack  these  elements. 

(3)  Many  required  data  elements  are  never  reported  outside 
of  base  level  or  contractor  plants. 

(4)  Some  units  report  data  elements  only  to  their  major 
using  command. 

(5)  AFM  66-1,  Maintenance  Data  Collection  System  (MDCS), 
requires  only  certain  data  from  AFTO  Forms  210/211 
to  be  reported  to  AFLC. 

(6)  During  the  Acquisition  Phase,  many  data  elements  are 
available  only  within  contractor  plants. 

Some  other  problems  require  elaboration.  First,  in  any  data 
system  one  must  minimize  the  administrative  or  clerical  burden  placed  on 
the  technician,  either  in  the  field,  or  at  the  factory  or  .test  site.  To  this 
end,  various  systems/procedures  were  examined  to  determiQe^the  most 
effective  yet  economical  method  in  both  cost  and  manpower,  formas suring 
that  all  necessary  and  accurate  data  are  obtained  from  the  field.  Thd' 
approach  which  appears  most  feasible  and  promising  to  solve  the  short  term 
acquisition  problem  is  assignment,  by  the  command  having  engineering 
responsibility,  of  a  systems  data  specialist  equipped  with  microfilm  equip¬ 
ment  to  using  command  bases  employing  selected  weapon  systems.  His 
function  would  be  to  obtain  a  microfilm  record  of  all  pertinent  data  produced 
in  support  of  the  specific  weapon  system  and  periodically  submit  for 
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processing,  storage  and  analysis  the  film  record  to  a  designated  informa¬ 
tion  center,  such  as\he  System  Effectiveness  Information  Central  (SEIC)  or 
System  Information  Bank  (SIB),  which  are  described  in  detail  in  this  report. 
Validity,  accuracy,  and  completeness  of  data  could  be  determined  in  a 
timely  manner  by  this  individual  on  site.  Aanajor  problem  presently  existing 
relative  to  error  audit  and  resultant  loss  of  important  data  and  time  from 
omissions  on  submitted  data  would  thus  be  eliminated.  The  procedure  out¬ 
lined  above  would  also  provide  a  means  of  closing  the  loop  on  raw  data 
retrieval  and  dissemination,  an  action  which  presently  is  almost  nonexistent. 
Further,  this  would  provide  the  means  for  indefinite  retention  of  that  data 
considered  pertinent  for  selected,  or  designated,  weapon  systems. 

Another  problem  is  that  the  present  Air  Force  and  major 

command  directives  and  procedures  now  permit  no  flexibility  in  records 

1 

retention;  e.g.,  field  organizations  are  authorized  to  destroy  AFTO  Forms 
210,  211,  212,  etc.,  after  a  period  anywhere  from  two  to  six  months.  When 
one  recognizes  that  only  a  portion  of  the  data  generated  and  recorded  on 
AFTO  Forms  210/211  is  transmitted  to  the  AFLC  AFM  66-1  data  repository, 
it  follows  that  much  potentially  valuable  and  pertinent  information,  from  the 
viewpoint  of  the  developing  command,  is  not  retained  for  the  life  of  a  weapon 
system. 

Finally,  the  problem  of  adapting  the  AFM  66-1  Maintenance 
Data  Collection  System  (MDCS)  to  provide  all  required  system  effectiveness 
data  and  information  has  been  considered.  Many  different  government 
agencies,  groups,  and  committees  are  presently  attempting  to  align  or 
adapt  their  data  collection  systems  to  AFM  66-1.  Headquarters  USAF  had 
designed  and  operated  the  AFM  66-1  MDCS  for  a  specific  and  limited  pur¬ 
pose;  i.  e.,  to  provide  maintenance  man-hour  accounting  and  logistics 
information  to  base  maintenance  management  and  AFLC.  In  these  areas  it 
has  been  a  step  in  the  right  direction  and  a  needed  service.  However,  it  is 
considered  neither  practical  nor  economical  to  reorient  this  program  solely 
as  n  system  effectiveness  information  system.  Rather,  the  Air  Force  should 
continue  to  use  it  as  initially  conceived,  with  modifications,  and  incorporate 
it  into  the  System  Effectiveness  Information  Central  as  one  of  the  many  system 
data  banks  or  information  centers  to  permit  drawing  upon  its  capabilities. 
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■?.  Summary  of  Section  on  Data  Central  System 

With  the  increasing  complexity  cf  modern  systems,  the  need  for 
system  effectiveness  information  has  also  become  more  critical.  Special¬ 
ized  information  systems  have  been  devised  to  respond  to  these  data  needs, 
but  in  general,  they  have  been  inadequate  and  particularly  unresponsive  to 
even  minor  changes  in  data  requirements.  The  Air  Force  information 
system  can  perhaps  be  classified  as  a  changing  family  of  individual  systems, 
each  collecting  and  processing  data  in  an  individual  way,  rather  than  as  a 
single  integrated  system.  Information  provided  by  those  systems  is  pri¬ 
marily  for  use  by  those  agencies  for  which  the  systems  were  designed. 


At  the  same  time,  the  Air  Force  information  requirements  make 
unreasonable  demands  on  these  unintegrated  systems.  The  two  main  prob¬ 
lems  are  (1)  flexibility  of  type  of  information  and  (2)  speed  of  response. 

A  solution  to  these  problems  is  a  System  Effectiveness  Information  Central 
(SEIC)  controlling  one  or  more  System  Information  Bank(s)  (SIB's)  into  which 
raw  data  are  fed.  The  purpose  of  the  SEIC  would  be  to  define  data  collection 
terms,  products  produced,  agencies  responsible  for  collecting,  processing 
and  retaining  data,  and  the  methodology  and  equipment  used  in  the  system. 
Communications  entry  into  the  various  agencies  and  their  data  repositories 
(hereinafter  referred  to  as  system  data  banks  or  information  banks)  is 
required.  The  SEIC,  in  essence,  would  maintain  a  comprehensive  index  of 
pertinent  information  and  have  the  communications  wherewithal  to  extract 
data  from  the  various  system  data  banks.  These  data  would  be  consolidated 
into  periodic,  one-time  or  special  system  effectiveness  reports  for  use  by 
various  levels  of  management. 


Maximum  utilization  should  be  made  of  present  and  future  com¬ 
puter  system  techniques  to  process,  analyze,  and  produce  desired  reports 
for  any  Air  Force  or  industrial  agency  authorized  access  to  the  information. 
The  ultimate  goal  would  be  an  all-knowledgeable,  all-responsive  source  of 
factual  information  concerning  the  many  disciplines  bearing  on  system 
effectiveness.  The  desired  information  should  be  available  either  within 
the  SEIC  central  repository  or  the  System  Information  Banks. 
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To  attain  the  ultimate  goal,  a  two-step  approach  is  recommended. 

i 

First,  a  pilot  phase,  SEIC  I,  would  be  applied  against  a  selected  weapon 
system;  o.g.,  Minuteman  Wing  VI.  The  pilot  phase  would  consolidate  the 
procedures  of  agencies  presently  engaged  in  diverse  but  closely  related  data 
retrieval  and  processing  actions  in  support  of  the  selected  weapon  Bystem, 
and  "debug"  the  proposed  methodology  and  techniques.  Concurrently,  the 
second  phase,  SEIC  II,  would  start  and  would  develop  procedures  for  appli¬ 
cation  on  a  larger  and  more  comprehensive  scale  on  future  systems. 

The  agency  established  to  operate  SEIC  II  should  be  a  pure 
"service"  organization,  established  to  provide  system  effectiveness  infor¬ 
mation  service  to  all  USAF  commands  and  aerospace  industries.  Its  only 
mission  in  life  would  be  the  processing  of  system  effectiveness  information. 
Conceived  as  initially  being  under  the  operation  and  control  of  AFSC,  SEIC  II 
could  be  operated  under  the  control  of  DOD  should  it  be  desired  to  expand 
the  system  to  include  other  military  and  civilian  agencies.  Much  detailed 
effort  remains  to  bring  to  fruition  this  proposal.  However,  the  principles 
and  techniques  presented  are  within  the  "state-of-the-art." 

4.  Summary  of  Section  on  Data  Management  Reports 

Information  derived  from  review  of  various  USAF  and  major 
command  management  reports  requirements,  discussions  with  industry  and 
Air  Force  major  command  representatives,  and  discussions  within  WSEIAC 
imply,  perhaps  erroneously,  that  both  industry  and  Air  Force  management 
have  in  the  paBt  concentrated  primarily  on  only  two  types  of  status  reporting 
in  the  design  and  development  of  weapon  systems;  i.e.,  cost  and  schedule. 
This  concentration  has  highlighted  such  techniques  as  PERT,  PERT  Cost, 
Cost  Planning  and  Appraisal,  etc.  While  cost  and  schedule  deserve  this 
attention  in  both  Air  Force  and  industry,  there  remains  an  urgent  need  to 
develop  similar  control  over  the  area  of  technical  performance,  not  only  in 
design  and  development  (Acquisition  PhaBe),  but  also  in  the  operating  envi¬ 
ronment  (Operational  Phase)  of  the  system  life-cycle.  Technical  perform¬ 
ance  seemingly  becomes  of  major  concern  only  at  such  times  as  inadequate 
performance  factors  (i.e.,  hardware  reliability,  CEP,  etc.)  occurs  relative 
to  the  estimated/pr ejected  figure  of  merit  at  the  particular  time  in 
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question,  or  following  turnover  of  an  operational  element  of  the  weapon 
system  lu  Ihe  using  command.  It  is  recognized  that  a  continual  interplay  of 
cost,  schedules,  and  performance  activity  exists  and  must  be  displayed  in 
an  integrated  fashion  in  any  management  reporting  media. 

From  the  development  standpoint,  hardware  reliability  to  the 
exclusion  of  other  supporting  disciplines,  alone  appears  to  have  been  the 
major  criterion  upon  which  the  degree  of  effectiveness  has  been  measured. 
From  a  using  command  viewpoint,  in-commission  or  availability  and  capa¬ 
bility  to  successfully  countdown  to  launch-commit  appear  to  be  the  major 
criteria  for  measuring  effectiveness. 

In  summary,  only  recently  has  Air  Force  management  concerned 
itself  with  a  simultaneous  review  of  all  elements  essential  to  assessing  sys¬ 
tem  effectiveness.  Further,  industrial  management  has  been  primarily 
interested  in  developing  those  techniques  which  the  Air  Force  has  stressed 
in  order  to  respond  effectively  to  one-time  and  recurring  data  requirements 
or  schedule  position. 

With  this  background  in  mind,  Task  Group  III  has  approached  the 
management  reporting  format  objective  from  the  viewpoint  of  compiling  as  a 
.cjo/isoJidatjed  report  --  for  comparison  purposes  --  not  only  major  cost  and 
schedule  information  but  also  the  key  characteristics  comprising  figures  of 
merit  for  system  effectiveness.  It  is  the  interaction  and  interrelationship 
of  these  figures  of  merit  that  have  an  impact  on  whether  the  weapon  system 
and  its  subelements  are  measuring  up  to  the  initial  SOK  and  revised  SPP 
requirements  at  any  point  of  time  in  the  life-cycle. 

The  management  status  reports  devised  by  Task  Group  III  and 
displayed  in  Section  V  and  Appendix  III  allow: 

•  two  levels  of  detail: 

•  total  system  and  subsystem,  and 
■  within  each  subsystem  by  hardware  end  items; 

•  narrative  explanations  where  indices  regress  from  the  pre¬ 
vious  report; 

•  trend  charts;  e.g.,  reliability  growth  curves. 
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The  System  Program  Office  and  contractor  would  execute  all 
reports  through  the  Acquisition  Phase,  while  the  operating  command  would 
provide  all  information  during  the  Operational  Phase.  Since  the  intent  is 
not  to  eliminate  present  reporting  requirements,  managers  would  usually 
require  the  kind  and  amount  of  information  now  being  supplied.  The  report 
would  extract  and  compile  pertinent  (top  layer)  information  into  an  overview 
of  the  entire  weapon  system  cost,  schedule,  and  technical  performance 
factors  for  ease  of  visibility  and  comparison  at  any  level  of  management 
fie  sired . 

Initial  progress  has  been  made  in  identifying  key  system  effec¬ 
tiveness  figures  of  merit.  However,  additional  study  is  required  to  identify 
comprehensively  the  total  package  for  use  of  top  management.  Some  char¬ 
acteristics  already  identified  and  deemed  essential  may  eventually  be 
replaced  by  items  considered  more  critical.  These  proposed  reports  are 
compatible  with  the  new  Materiel  Program  Codes  (MPG)  used  in  current 
budget,  program,  and  accounting  procedures,  and  thus  provide  a  common 
thread  of  data  from  conception  through  operation  of  the  system. 

Because  these  management  reports  represent  a  wide  and  diverse 
collection  of  information  from  both  contractor  and  government  sources,  it 
may  be  impractical  to  grant  all  contractors  the  full  scope  of  such  reports. 
Thus,  the  principle  would  be  established  to  provide  each  contractor  only  - 
that  information  pertaining  to  his  own  activity.  The  Materiel  Program  Codas 
are  especially  valuable  in  this  connection  because  they  are  structured  at  the 
first  level  of  indenture  by  prime  and/or  associate  contractors. 


D.  TASK  GROUP  IV,  "COST-EFFECTIVENESS  OPTIMIZATION" 

VOLUME  I 

SUMMARY,  CONCLUSIONS,  RECOMMENDATIONS'0 

I .  Abstract 

The  underlying  principles  associated  with  cost-effectiveness 
analysis  arc  discussed.  The  rationale,  purpose,  methodology  required, 
and  nature  of  the  results  that  can  be  obtained  by  means  of  the  analysis  are 
presented  in  summary  form.  Illustrations  of  the  type  of  input  data  required 
and  the  logic  associated  with  its  application  are  provided.  The  summary 
constitutes  an  overview  of  the  more  detailed  task  analysis  and  supplementary 
technical  material  presented  in  Volumes  II  and  III,  Included  are  the  con¬ 
clusions  and  recommendations  as  set  forth  in  Volume  II. 

VOLUME  II 

TASKS  AND  ANALYSIS  METHODOLOGY' ' ' 

1 .  Abstract 

The  report  discusses  the  philosophy  of  cost-effectiveness  and 
techniques  for  trade-off  and  optimisation  Btudies.  It  lists  and  discusses 
twelve  tasks  necessary  to  perform  a  cost-effectiveness  analysis.  A 
methodology  is  outlined  for  identifying  and  standardizing  cost  and  effective¬ 
ness  factors,  Descriptive  analytical  models  for  cost-effectiveness  are 
provided,  including  discussion  of  their  sensitivity  and  validity.  One  section 
defines  and  discusses  risk  and  uncertainty  and  their  effect  on  the  decision 
making  process.  Included  is  an  extensive  bibliography  on  coBt-effectiveness. 
Examples  of  some  of  the  techniques  are  covered  in  detail  in  a  "Technical 
Supplement,"  which  is  Volume  III  of  the  final  report  of  Task  Group  IV. 
Abstracts  of  these  examples  will  be  found  in  the  Appendix  of  this  report. 

2.  Overview 

A  major  management  goal  throughout  the  life-cycle  of  a  system 
--  from  the  Conceptual  Phase  through  the  Operational  Phase  --  is  to  exer¬ 
cise  management  control  for  the  purposes  of  selecting,  developing,  and 
using  systems  in  an  optimum  manner.  The  process  bywhich  management 


is  provided  inputs  for  these  types  of  decisions  has  been  commonly  called 
cost-effectiveness  analysis. 

Effectiveness  is  a  measure  of  the  capability  of  the  system  to 
accomplish  the  mission  objectives.  Cost-effectiveness  studies  are  concerned 
with  achieving  a  combination  of  resource -use  and  attained  effectiveness  that 
is  best  according  to  a  selected  criterion.  Resource-use  represents  the  ex¬ 
penditure  of  dollars,  manpower,  material,  time,  etc,,  required  for  the 
development,  operation,  and  support  of  a  system.  We  shall  interpret  such 
studies  as  an  attempt  to  quantify  how  much  it  costs  to  achieve  a  certain 
effectiveness  in  order  to  select  among  a  set  of  alternatives. 

There  is  a  recognized  need  for  such  studies.  The  enormous 
responsibility  of  the  Department  of  Defense  and  the  military  services  for 
maintaining  a  strong  posture  involves  considerable  expenditure  of  national 
resources.  This  is  clearly  evidenced  by  the  proportion  of  the  federal  bud¬ 
get  now  allocated  to  defense,  It  is  thus  mandatory  that  the  military  authori¬ 
ties  exercise  maximum  control  in  their  planning,  procurement,  and 
operational  activities  in  order  to  minimize  the  burden  placed  on  the  economy 
without  any  sacrifice  in  over -all  defense  goals, 

Cost-effectiveness  analysis  is  not  new.  It  has  been  a  part  of 
military  planning  for  some  time,  but  the  complexity  of  the  military  tasks 
now  requires  a  multidisciplinary  approach.  The  major  utility  of  cost- 
effectiveness  analysis  is  to  provide  management  with  the  necessary  informa¬ 
tion  for  decision-making  purposes  utilizing  all  the  available  knowledge  and 
data  in  as  efficient  and  complete  a  manner  as  is  possible.  Consequently,  a 
demand  has  been  created  for  improved  analytical  methods,  better  and  more 
complete  data,  expanded  computational  capacity,  etc,,  which  has  improved 
and  will  continue  to  improve  management's  capability  for  making  good 
decisions. 

3,  Levels  of  Cost-Effectiveness  Analyses 

There  are  several  decision  making  levelB  at  which  a  cost- 
effectiveness  analysis  can  be  meaningfully  applied,  and  these  roughtly 
correspond  to  the  phases  during  system  development.  One  level  for 
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application  is  at  the  Required  Operational  Capability  (ROC)  level,  formerly 
called  the  General  Ojsferational  Requirement  (GOR)  phase.  The  ROC  estab¬ 
lished  a  spectrum  of  objectives  or  missions.  By  considering  over -all 
defense  goals,  the  geopolitical  and  environmental  factors,  and  the  economic 
and  technological  capabilities,  a  particular  rrrissiori  or  objective  is  selected. 
This  level  of  application  is  generally  coordinated  at  the  DOD  level. 

After  mission  requirements  are  set  at  the  Specific  Operational 
Requirements  (SOR)  phase,  there  exists  the  need  for  selecting  alternate  or 
competing  systems.  Application  of  cost-effectiveness  analysis  at  this  level 
is  primarily  the  responsibility  of  the  military  or  procuring  agencies. 

A  third  level  of  application  occurs  during  the  development  and 
operation  of  the  weapon  system.  This  lev^l  of  application  furnishes  infor¬ 
mation  for  optimal  use  of  resources  within  the  constraints  of  mission  and 
system  requirements. 

As  a  simple  example  of  these  levels  the  first  would  be  concerned 
with  such  problems  as  o'ptimum  force-mix;  e.g.,  expanded  bomber-force 
size  versus  expanded  missile-force  size.  The  second  level  would  be  con¬ 
cerned  with  such  combinatorial  choices  as  pertain  within  a  class  of  systems; 
e.g.,  within  missile  systems  we  may  examine  liquid  versus  solid  fuel,  tan¬ 
dem  versus  parallel  stages,  or  soft  versus  hardened  sites.  The  third  level 
would  be  concerned  with  more  detailed  decisions  within  a  given  system  con¬ 
figuration;  e.g.,  for  a  missile  one  might  evaluate  pressurized  or  pump-fed 
propellant -loading  systems,  various  stage  diameters,  various  area  ratios 
of  engine  nozzles,  checkout  and  monitoring  procedures,  and  the  like. 

This  report  is  concerned  primarily  with  the  third  level  of  analysis 
and,  to  a  lesser  extent,  with  the  second  level,  in  presenting  and  illustrating 
the  concepts,  methods,  and  procedures  of  cost-effectiveness  analysis. 

4 .  General  Concepts 

To  introduce  the  general  concepts  of  a  cost-effectiveness 
analysis,  we  shall  interpret  such  analysis  in  the  simplest  of  terms  --  namely, 
the  attempt  to  quantify  how  much  it  costs  to  achieve  a. certain  effectiveness  in 
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order  to  select  among  c^set  o£  alternatives.  Cost  is  used  to  represent  the 
amount  of  resource  expenditure,  and  effectiveness  is  a  measure  of  the 
system  ability  to  accomplish  its  mission  objectives. 

The  general  approach  for  making  such  decisions  consists  of  the 


following  steps: 


(1)  Define  criteria  for  selection 

(2)  Generate  alternatives  that  satisfy  operations 
requirements  and  constraints 


(3)  Compute  resultant  values  of  cost  and  effectiveness 


for  each  alternative 


Evaluate  results  with  respect  to  the  decision 
criterion.  * 


-  '"W 

■ 


Each  of  these  major  steps  is  discussed  in  detail  in  the  report. 
It  is  worthwhile,  however,  to  set  the  stage  for  such  discussions  in  this 
introduction.  >. 


The  criterion  for  selection  must  be  one  that  is  mission  respon¬ 
sive,  that  is,  it  must  answer  the  right  question.  Essentially,  the  criterion 
is  based  on  maximizing  effectiveness  for  a  given  cost  or,  conversely,  mini¬ 
mizing  cost  for  a  given  level  of  effectiveness.  The  criterion,  however, 
must  also  define  the  level  of  analysis  as  discussed  previously  in  this  intro¬ 
duction  and  also  the  scope  of  the  analysis  in  terms  of  resource,  system, 
operational,  and  support  constraints.  Thus,  the  two  basic  criteria  listed 
above  mav  evolve  into  a  criterion  such  as  one  to  maximize  effectiveness  per 
dollar,  provided  effectiveness  is  greater  than  E*  and  cost  is  less  than  C* 
(where  E*  and  C*  refer  to  specific  limiting  values). 


In  generating  acceptable  alternatives,  identification  of  all 
variable  and  fixed  factors  and  their  costs  is  required.  In  addition,  the 
elements  of  risk  and  uncertainty  as  related  to  these  factors  and  costs  and 
the  analysis  of  effects  on  other  programs  must  also  be  considered.  Such 
factors  as  availability  of  appropriate  data,  computational  capacity,  and 
restraints  in  time  and  effort  available  for  the  analysis  will  play  important 
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roles  in  this  phaseV  A  generated  alternative  is  then  an  acceptable  combina¬ 
tion  of  the  selected  factors  with  associated  risk  and  uncertainty  elements. 

Measures  of  cost  and  effectiveness  for  each  design  alternative 
must  then  be  computed.  The  form  these  measures  take  is  related  to  the 
decision  criterion.  For  effectiveness,  the  measure  can  range  from  a  sim¬ 
ple  probability  numeric,  to  an  expected  value,  to  the  complete  distribution 
of  some  over -all  performance  characteristic.  The  effectiveness  model  is 
based  on  sub-models  for  reliability,  maintainability,  and  performance. 
These  in  turn  are  based  on  the  variable  arid  fixed  factors  to  be  considered 
such  as  failure  and  repair  distributions,  internal  stresses,  environment, 
and  design  integration. 

The  cost  measure  must  be  one  that  can  treat  the  major  typ'es  of 
resource  expenditures  on  some  common  basis.  Sub-models  are  required 
for  development  costs,  operating  costs,  and  support  costs  both  in  terms  of 
dollars  and  schedules,  In  addition,  the  burden  a  particular  alternative 
places  on  other  systems  and  objectives  must  be  evaluated  for  a  complete 
cost  model. 

The  integration  of  the  separate  cost  and  effectiveness  models 
into  a  single  cost-effectiveness  model  provides  the  basis  for  decisions.  It 
is  at  this  stage  where  optimization  theory  becomes  applicable,  involving 
such  disciplines  as  mathematical  programming,  stochastic  process  theory, 
calculus  of  variations,  econometrics,  and  decision  theory. 

All  of  the  above  models  must  satisfy  characteristics  related  to 
adequacy,  representativeness,  consistency,  sensitivity,  plausibility, 
criticality,  workability,  and  suitability  These  characteristics  are  dis¬ 
cussed  more  fully  in  later  sections.  In  applying  the  model,  it  must  be 
emphasized  that  results  of  the  optimization  process  can  only  indicate  the 
best  decision  within  the  simplifications,  assumptions,  restrictions  and 
omissions  that  were  required  to  circumvent  such  problems  as  uncertainties, 
non-quantifiable  factors,  and  inadequate  data,  time  or  computational 
capacity. 

Thus,  the  cost-effectiveness  analysis  will  usually  yield  only 
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partial  analytic  solutions.  ..However,  the  framework  for  a  final  decision  is 
provided.  The  cdst-effectiveness  analysis  has  reduced  the  guess  work  and 
intuitive  estimates  of  cost  and  effectiveness,  but  the  initial  results  must 
still  be  critically  evaluated  and  combined  with  relevant  political  and  timing 
factors  by  a  judgment  of  the  decision  maker. 

VOLUME  III 

TECHNICAL  SUPPLEMENT  (8) 

1,  Abstract 

V  * 

A  discussion  of  optimization  is  presented  which  amplifies  the 
material  in  Volume  II,  Section  IV.  Optimization  principles,  criteria  and 
checklists,  as  well  as  a  summary  of  various  applicable  techniques  is 
included.  A  series  of  six  examples  are  described  covering-^  number  of 
critical  aspects  of  cost-effectiveness  analysis  in  considerable  detail.* 

Treated  in  the  examples  are:  (1)  Optimization  of  effectiveness  based  on 

* 

reliability,  maintainability,  performance,  and  cost;  (2)  Allocation  of 

,  t- 

reliability  requirements  among  subsystems;  (3)  Payload  allocation  among 
three  subsystems  based  on  a  fixed  weight  constraint;  (4)  Determination  of 
best  checkout  routine  for  a  duration  constrained  pre -launch  test;  (5)  Opti¬ 
mization  of  availability  for  a  system;  and  (6)  Trade-off  study  between  site 
hardening  and  dispersal  for  a  missile  system. 

2.  Summary 

The  examples  presented  in  Volume  III  of  the  Task  Group  IV 
report  are  summarized  below. 

Aircraft  System  Optimization  A  system  cost-effectivcncss 
model  is  developed  for  an  Air  Force  training  base  at  which  daily  bomber 
training  flights  are  made.  In  the  event  of  enemy  attack,  the  base  bomber 
force  is  assigned  to  targets.  The  objective  of  the  example  is  to  illustrate 
the  optimization  of  the  bomber  effectiveness  by  trading  off  reliability,  main 
tainability,  performance  and  cost  factors.  The  system  effectiveness  model 
is  developed  along  the  mathematical  lines  presented  by  Task  Group  II  in 

(3)  _  . 

Volume  II  of  their  final  report.  Optimization  is  accomplished  by  com¬ 
puting  and  comparing  the  costs  of  eight  possible  measurement  and  support 
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policies  in  terms  of  two  alternative  figures  of  merit. 

•  For  <hfc£h  target,  there  will  be  a  0,  95  probability  that 
at  least  ime  of  the  attacking  aircraft  will  successfully 
accomplish  the  bombing  run. 

*  *  :  >  i  —  ■  ‘ 

•  There  will  be  an  average  success  probability  of  0.95 

for  all  assigned  targets. 

A  significant  aspect  of  this  example  is  its  illustration  of  the 
need  for  re-evaluating  the  criterion  for  optimization  in  terms  of  the  realized 
output  of  the  evaluation  effort. 

Reliability  Allocation  A  method  for  allocating  system  reliability 
requirements  among  subsystem  (or  lower  level  units)  is  presented.  The 
method  considers  serial  and  redundant  interconnections  a m o ifg~f )tp~ sub 

■  |  ■■■  ’■ 

systems.  The  relationship  between  system  reliability  requirements  and  y 
system  effectiveness  requirements  is  considered. 

Ballistic  Missile  Payload  Allocation  Each  element  of  a  ballis- 

. -  —  *"  rr  - ■  . -  . . 

tic  missile's  payload  --  warhead,  guidance  and  penetration  aids  --  will 
'  increase  in  effectiveness  with  an  increase  of  weight  allocated  to  the  element. 
For  a  missile  that  is  to  be  employed  against  a  defended  "point"  target,,  this 
example  presents  a  method  for  determining  the  optimum  division  of  the 
missile's  payload  between  the  three  competing  (for  weight)  elements,  when 
their  individual  weight-effectiveness  relationships  are  known.  For  the  case 
•if  a  single  missile  per  target,  Using  a  most  basic  application  of  the  step- 
wise  opt  imization  philosophy  of  dynamic  programming,  the  problem  is 
formulated  as  a  two-stage  weight  allocation  process.  The  first  stage  deter¬ 
mines  th>-  optimum  trade-off  between  warhead  (lethal  radius)  and  guidance 
(f'.F.l*);  the  second  stage  determines  the  optimum  division- between  penetration 
!  i'i;-.  and  m  optimum  mix  of  warhead  and  guidance.  The  same  optimization 
proi  >-ss  is  useful  for  the  cases  of  sequential  and  simultaneous  muUjjde 
mis.  ile  employment  per  target*  Although  this  design  opt im izalion  ’probl em 
>  ; i ,  In  solved,  fun*  t  iou.tllv,  for  the  modes  of  missile  employment  considered, 
't  i  !>}>li*  ability  to  i  r  •  ■  a  la  I  leu  a  t  ion  problem  is  confounded  by  the  design, 

.tit*  t'K  •em-e  *nd  •■mp.loy nv-nt  estimates  required  in  the  analysis.  Use  of  this 
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method  c  ould  show,  however,  the  influence  of  the  estimated  uncertainties  on 
the  optimal  payload  division  atic^could  thereby  serve  as  a  useful  point -of 
departure  for.  design  cOrriprorriises .  • ' '  v.' : * •  . 

Optimising  a  Pre -Launch  Checkout  This  example  presents  a 
procedure  for  determining  the  optimum  test  content  of  an  1CBM  pre  -launch 
checkout  that  is  subject  to  a  time  constraint.  Cost  considerations  axe  not 
introduced  as  a  constraint,  b<it  instead  arc  employed  after  the  test  content 
h  .s  optimized  for  each  possible  test  duration  constraint  in  order  to 

-ot  liirtwiM’ii  designs.  An  example  is  given  and  references  are  cited .that 
..nl.i  in  ah  ..  xplaiiatioh  of  the  estimation. of  the  parameters  associated  with 
i  h t •  design  technique.  .  • 

Missile  Availability  The  availability  of  a  system  subjected  to 
.  quvm  v  of  calendar  Spaced  checkouts  is  considered.  Formulae  for  cal-: 

-  n ! .d inji  lb.  optimum  frequency  of  checkout  arc  given  for  the  situation  which 
.ni.'r.:  .  heckout  time  as  down  time.  Imperfect  repair;  imperfect  check - 
..*,<[  resource  limitations ’are  treated.  A  technique  for  the  estimation  of 
•  ••  ,.  r  -CO'  ler>:  of  the  availability  nriOdcl  is  also  givetV. 


E.  TASK  GROUP  V,  “MANAGEMENT  SYSTEMS1* 

•>'.  VOLUME  I  • 

SYSTEM  EFFECTIVENESS  ASSURANCE  -  .  1  . 

SUMMARY,  POLICY  ISSUES,  RECOMMENDATIONS'9'  . 

I,  Abstract  . 

A  system  effectiveness  assurance  management  concept  is  pre¬ 
sented.  It  is  based  on  the  interacting  process  of  experience  retention 
(resources  development)  and*  program  management  technology  (resources 
application)..  Six  segments  integral  to  this  process  are  identified:  data 
acquisition,  technology  development,  personnel  development,  program 
planning,  input  surveillance,!  and  output  evaluation.  The  recognition  and 
disciplined  treatment  of  activities  and  elements  critical  to  effectiveness 
included  within  each  of  the  six  segments  of  effectiveness  assurance  constitute 
(In*  management  system.  Current  practice  of  Air  Force  arid  industry  is 
assessed,  a  group  of  fourteen  basic  policy  issues  are  raised  along  with 
related  recommendations,  and  ah  implementation  plan  is  described. 

2.  Introduction  and  Overview 

*u  Scope  A  concept  and  philosophy  of  system  effectiveness 
assurance  management  has  been  developed  by  Task  Group  V.  On  the  basis 
<»f  discussions,  surveys  and  critical  examination  of  the  present  status  of 
effectiveness  management  throughout  the  Air  Force  and  industry,  the  prin¬ 
cipal  elements  have  been  identified,  policy  issues  have  been  raised,  arid  an 
implementation  plan  recommended. 

It  is  recognized  that  effectiveness  must  inevitably  be 
treated  as  a  quantitative  characteristic  of  systems.  Technical  "requirements, 
measurement  methods,  data  elements,  and  cost-effectiveness  analysis  are 
essential  ingredients.  However,  these  aspects  arc  treated  fully  in  the 
r.  ports  r»f  Task  Groups  I,  II,  III  and  IV.  Task  Group  V  has  attempted  to 
di  s*  r »!>••  a  management  system  that  can  form  the  basis  for  a  neW  focus  in 
Hie  ,\irt  Force  and  .industry  through  which  major  improvements  in  effcjctive- 

eiv.s  «  an  be  achieved.  "i 
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b.  Emphasis,  on  People  Task  Group  V's  general  objective 

was  to  improve  the  system  effectiveness  assurance  aspects  of  the  Air  Force 
management  system.  The  purpose  of  any  management  system  is  to  get 
things  done  through  people.  Consequently,  this  Task  Group  has  endeavored 
to  identify,  develop  and  recommend  methods  for  program  management  which 
will  assure  the  achievement"  of  system  effectiveness  objectives,  An  Air 
Force  definition  states  that:; 

"Management  is  the  process  of  developing 
and  applying  resources  to  accomplish  pre¬ 
determined  objectives." 

This  definition  was  adopted  and  used  to  further  refine  the  Task  Group  V 
objective.  The  objective  thus  became  "to  help  improve  the  Air  Force 
resources  development  program  and  resources  application  requirements 
relative  to  those  activities  of  people  that  are  most  critical  to  achieving 
system  effectiveness."  Emphasis  has  been  placed  on  identifying  qualified 
people  as  the  primary  resource  for  achieving  system  effectiveness.  The 
importance  of  improving  communications  between  the  many  groups  whose 
skills  must  be  utilized  to  achieve  an  effective  weapon  system  has  been 
stressed- 

It  was  recognized  that  sanctimonious  generalities  about 
capable  people,  good  communications  and  able  management  would  not  help. 
Therefore,  strenuous  efforts  were  made  to  be  specific.  The  term  "System 
Effectiveness  Critical  Activity"  (SECA)  was  introduced  and  defined.  A 
SEGA  is  any  activity  that  experience  has  shown  must  be  subjected- to  formal 
discipline  in  order  to  ensure  .system  effectiveness. 

The  term  "discipline"  is  very  important  but  subject  to 
many  connotations.  To  avoid  misunderstanding,  the  Task  Group  equated 
discipline  with  all  types  of  control,  over  the  activities  of  people.  Specifically, 
it  t  onsists  of: 

( 1 )  training 

(2)  motivation 
(  >)  command 
(•1)  audit 
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Again,  the  Task  Group  recognized  that  general  comments  on  these  four  types 
of  c  ontrol  over  activities  of  people  would, not  improve  the  system  effective¬ 
ness  assurance  aspects  of  the  Air  Force  managment  system.  However, 

J  application  of  these  general  principles  to  specifically  identified  system 
■  effectiveness  critical  activities  allows  specific  evaluation  of  current  status 
and  defines  those  area  where  Specific  improvement  actions  can  be  introduced. 

:  For  example,  an  Air  Force  program  director  may  receive  general  training- 
in  program  management  philosophy  but  still  be  left  in  doubt  about  specific 
actions  that  lie  should  take  in  managing  a  new  program.  By  contrast,  if  he 
is  taught  that  there  is  a  tangible  activity  called  "functional  flow  analysis,"  - 
if  he  is  provided  with  documented  technology  for  this  activity  and  motivated 
tu  apply  it,  --  if  he  is  commanded  by  AFSGM  375*5  to  require  his  contrac¬ 
tors  to  schedule  and  fund  the  performance  of  this  activity,  --  and  if  the 
program  is  audited  by  his  inspector  general,  there  is  a  high  probability  that  * 
Ihp' activity  will  be  accomplished. 

The  Task  Group  found  it  necessary  to  categorize  activities 
of  people  into  the  following  two  types: 

(1) 

(2) 


decision  creation  activities 
•  * 

hardware  creation  activities. 


To  assure  system  effectiveness,  it  is  necessary  to  assure  the  quality  of 
both  derision  making  and  hardware  making.  Assuring  the  quality  of  manual 
and  machine  activities  that  create  hardware  is  the  recognized  purpose  of 
AF/Industry  Quality  Control  management  systems,  and  Task  Group  V  did 

•  onrern  itself  with,  and  its  recommendations  do  apply  to,  assuring  the  quality 

•  if  hardware  creation  activities.  However,  the  group's  dominant  effort  was 
1  • »  <1.  fine  decision  making  critical  activities  and  to  contribute  to  resources 
<!••;.  ••.lopment  and  resources  application  requirements  for  these  activities. 

r.  Experience  Retention  For  most  critical  activities,  the 
i  fi.i.i  •  rv  siiiiri  !-  of  knowledge  is  the  operating  experience  of  the  Air  Force 
•••:>  .  .'>nl  r.n  tors.  !l  follows  that  resources  development  for  these  activi- 

. .  f  •  -r.ploil  this  experience.  Task  Group  V  identified  and  defined 
1  •  in  «  ■•nve  rl  ing  experii-nre  into  resources  suitable  for  application 
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to  future  programs.  These  steps  are: 

(1J  Data  Acquisition 

(2)  Technology  Development 

(3)  Personnel  Development. 

Collectively,  these  three  steps  constitute  System  Effectivene s s  Experienc e 
Retention.  " 

d.  Program  Management  Technology  The  resources  - 
application  segment  of  the  Air  Force  management  system  was  identified 
with  the  procedures  set  forth  in  the  AFSC  375  series  documents.  The  term 
"program  management  technology"  was  introduced  to  represent  all  the  for¬ 
mal  disciplines  for  assuring  effective  application  of  resources  for  each 
activity  critical  to  effectiveness.  The  basic  steps  in  program  management 
techn  >1  igy  were  identified  asi 

(1)  Program  Planning 

i.(2)  Input  Surveillance 

(3)  Output  Evaluation. 

Combination  of  these  three  steps  with  the  three  previously  identified  results 
in  a  six  segment  management  system.  The  term  "System  Effectiveness 

Assurance  Management  System"  (SEAMS)  was  introduced.  K* 

e. .  .  Status  of  Critical  Activities  Volume  II  of  the  Task 
Group  V  final  report  presents  a  series  of  surveys  and  discussions  on  the 
status  of  effectiveness  management' in  the  Air  Force  and  industry.  Addition¬ 
al  discussions  of  some  of  the  more  significant  aspects  or  elements  of 
effectiveness  assurance  are  also  included.  These  surveys  and  discussions 
arc  arranged  in  general  around  the  six  formal  segments  contained  in  the 
broad  management  concept  Of  Experience  Retention  and  Experience  Applica¬ 
tion  listed  in  paragraphs  c  and  d  above.  A  summary  of  Volume  II  is 
contained  in  Section  II  of  the  report. 

f  Policy  Issues  During  the  course  of  the  detailed  investi¬ 

gations  and  studies  of  the  Task  Group,  a  number  of  fundamental  policy  issues 
•were  identified.  Appropriate  consideration  of  these  issues  is  basic  to 
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further  progress  in  the  development  of  both  technical  and  management  u 
aspects  of  system  effectiveness.  These  policy  issues  are  described  tin 
Section  IV,  and  are  the  basis  upon. which  specific  recommendations  and 
suggested  implementation  plans  are  developed  in  Section  V. 

g.  Recommendations  and  Implementation  Plan  Although 
recommendations  occur  in  many  of  the  individual  status  reports  and  discus¬ 
sions  of  critical  activities  contained  in  Volume  II,  these  have  been  summar¬ 
ized  and  in  some  cases  generalized  to  form  a  consolidated  group  that  are 
related  to  the  major  policy  issues  and  to  the  six  formal  segments  of  the 
System  Effectiveness  Assurance  Management  System. 

The  Task  Group  did  wish  to  assist  the  Air  Force  by  doing 
some  of  the  work  proposed  in  the  Recommendations  Implementation  Plan. 
However,  such  work  could  not  be  started  until  broad  generalities  and  diver¬ 
gent  opinions  on  assurance  management  had  been  converted  into  a  concise, 
practical  management  system.  Moreover,  motivation  of  the  Task  Group 
members  to  implement  their  proposals  depends  on  a  thorough  and  positive 
evaluation  of  these  proposals  by  the  Air  Force.  It  is  important  to  recognize 
that  SEAMS  is  a  highly  integrated  system.  It  requires  positive  support  by  at 
least  Headquarters  USAF,  AFSC,  AFLC,  ATC  and  AU.  There  would  be 
little  point  in  Task  Group  members  contributing  parts  of  the  system  if  the 
whole  is  not  acceptable  to  these  Commands. 

The  Task  Group  recognizes  that  recommendations  are 
useless  unless  opposition  to  their  implementation  can  be  overcome.  Exten¬ 
sive  opposition  to  any  form  of  formalized  experience  retention  does  exist 
throughout  both  the  Air  Force  and  industry.  The  principal  bases  for  such 
opposition  appear  to  be: 

(1)  fear  that  lessons  learned  will  be  expressed  as 

rigid  procedures  that  will  restrict  the  exercise  of 
judgment  by  new  project  organizations,  and 

(Z)  the  conviction  of  project  managers  and  engineers 
that  "my  project  is  different." 


Task  Group  V  has  developed  and  described  logical  and  proven  methods  for  ,1, 
overcoming  both  these  types  of  opposition  to  formalized  experience  retenti8rij§||f 


VOLUME  II  >  nm 

ELEMENTS  OF  EFFECTIVENESS  ASSURANCE  MANAGEMENT'1^: 

•  ■  .  ■  . 

1.  Abstract  v  / 
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Volume  II  of  the  Task  Group  V  Final  Report  presents  summaries 
of  many  of  the  studies  carried  on  by  the  ThskGroup  during  the  six  months  of 
WSEIAC  formal  meetings  and  investigations.  From  these  studies  the  con-, 
cept  and  philosophy  of  a  System  Effectiveness  Assurance  Management  ,  ; 
System  (SEAMS), the  major  policy  issues,  and  recommendations  presented 
in  Volume  I  were  developed.  Air  Force  management  of  system  effective¬ 
ness  activities  is  assessed  through  surveys  of  some  of  the  principal- offices 
and  commands.  Industry  capability  for  response  to  these  new  requirements 
is  measured.  A  review  of  activities  and  discipline  requirements  essential 
to  effectiveness  assurance  management  in  the  context  of  the  AF  375  series 
documents  is  provided.  Finally,  a  series  of  studies  and  discussions  of  per¬ 
tinent  elements  of  system  effectiveness  is  furnished,  including  a  data  control 
proposal,  the  effect  of  incentives,  and  appendixes  on  specification  manage¬ 
ment,  parts  research  and  program  management  elements  of  the;  Conceptual 
and  Definition  Phases  of  system  development. 

2.  Introduction  and  Summary 


The  volume  presents  reports  covering  the  significant  investiga¬ 
tions  accomplished  by  Task  Group  V.  These  studies,  coupled  with  examina¬ 
tion  of  relevant  Air  Force  official  documents  and  research  reports  and  the 
many  discussions  held  by  task  group  members,  were  the  principal  basis  for 
development  of  the  effectiveness  management  philosophy,  policy  issues,  and 
recommendations  reported  in  Volume  T  of  the  Task  Group  V  report. 

a.  Status  Reports  -  Air  Force  Critical  Activities  Section  II 
summarizes  the  principal  findings  resulting  from  the  task  group's  appraisal 
of  a  number  of  Air  Force  organizations  whose  policies  and  practice  have  an 
overriding  effect  on  system  effectiveness.  With  the  time  available  for  the 
entire  study,  these  appraisals  could  not  possibly  delve  into  great  detail  on 


the  organizations  surveyed.  However,  there  was  a  consistent  thread  of 
information  that  became  apparent  as  the  interviews  continued.  It  is  believed 
that  the  findings  are  quite  representative  of  m^nt  comparable  segments  of 
the  Air  Force.  Where  possible,  the  appraisals  were  made  using  the  six 
formalized  segments  of  SEAMS  as  a  checklist.  Many  of  the  policy  issues 
and  recommendations  of  Volume  I  were  derived  from  these  investigations. 

b.  Indust/y  Attitudes  and  Climate  In  Section  III  of  the 
volume  a  number  of  recent  trends  in  industry  that  have  a  pronounced  effect 
on  system  effectiveness  are  examined.  Nine  (9)  specific  items  are  discussed 
including:  cost  reduction,  performance  evaluation,  weighted  guidelines, 

F  time  and  cost,  AF  375  series,  Air  Force  program  management, 
industry  surveys,  program  definition,  and  allowable  research.  These  re¬ 
cent  trends  cor  litute  a  climate  within  which  industry  is  now  operating. 

*  mally,  a  list  of  ten  (10)  guidelines  (and  pitfalls  to  avoid)  are  suggested 
that  have  been  found  to  be  useful  to  industry  in  adjusting  to  these  new  trends. 

c.  Elements  of  Effectiveness  Management  Section  IV  of 
the  volume  provides  a  rationale  for  identification  of  activities  critical  to 
system  effectiveness.  These  activities  and  their  associated  discipline 
requirements  form  the  backbone  of  SEAMS,  They  are  discussed  in  conjunc¬ 
tion  with  some  of  the  principal  elements  of  the  AF  375  series  documents. 

Six  steps  (which  constitute  the  Scientific  Method)  are  reviewed  as  a  suggested 
logical  process  for  effectiveness  analysis  and  problem  solution.  Some 
comments  on  organization  for  effectiveness  functions  are  given. 

d.  System  Effectiveness  Information  Central  Both  Task 
Groups  III  and  V  have  recommended  the  formation  of  an  Air  Force  central¬ 
ized  data  and  information  service.  The  proposals  of  both  groups  recognize 
the  need  for  a  consolidation  of  the  many  separate  data  systems  --  for  a 
central  bank  that  will  accumulate  pertinent  information  relative  to  system 
effectiveness  and  make  this  fund  of  knowledge  available  for  new  programs. 

Both  task  groups  have  considered  the  problem  from  a 
different  perspective,  and  yet  the  recommended  approaches  have  much  in 
common.  Task  Group  V  has  proposed  an  appropriate  set  of  objectives  for 


such  a  data  central,  its  proposed  scope,  organization,  program  «nri  folding. 
A  charter  is  also  suggested.  These  are  contained  in  Section  V. 

An  important  recommendation  of  Task  Group  V  (contained 
in  Volume  I)  is  that  a  careful  study  of  both  task  group  recommendations,  as 
well  as  presently  existing  data  centrals,  be  made  prior  to  implementation  of 
any  new  program. 

e.  Incentives  and  System  Effectiveness  Section  VI  contains 
some  commentary  on  the  new  incentives  now  appearing  in  Air  Force  con¬ 
tracts.  Some  of  the  possible  advantages  are  mentioned  along  with  the  many 
pitfalls  that  may  befall  the  government  and  the  contractor  if  incentive  mech¬ 
anics  are  not  carefully  worked  out  and  agreed  upon  prior  to  contract  signing. 
As  a  technical  requirement,  effectiveness  embodies  many  interrelated 
system  characteristics.  At  present  it  appears  that  attention  should  be 
directed  toward  some  of  the  principal  elements  of  effectiveness,  such  as 
reliability  requirements  or  service  warrantees,  rather  than  toward  the  com¬ 
posite  term. 

f.  Supplementary  Studies  and  Reports  The  appendix  con¬ 
tains  three  supplementary  items  pertinent  to  effectiveness  management. 
Appendix  I  contains  a  detailed  functional  flow  diagram  depicting  the  system 
program  management  elements  during  the  Conceptual  and  Definition  Phases. 
Because  of  the  early  stage  of  development  of  the  AF  375  series  and  subse¬ 
quent  pending  revisions,  Task  Group  V  did  not  propose  a  specific  revision  to 
include  elements  of  effectiveness  management  and  assurance.  However,  it 
is  obvious  that  the  format  now  emerging  offer b  a  natural  medium  by  which 
to  recognize  technical  and  management  aspects  of  effectiveness  assurancei 

Appendix  II  provides  guidelines  for  research  programs 
directed  toward  new  parts.  It  is  clearly  recognized  that  one  of  the  building 
blocks  of  an  effective  system  is  the  piece  parts  of  which  it  is  composed.  All 
too  frequently,  in  the  past,  major  program  delays  and  increased  costs  have 
resulted  from  use  of  an  unreliable  part  or  one  on  which  too  little  information 
is  known  to  allow  proper  application. 
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Appendix  III  describes  the  evolution  of  the  reliability 
specification,  Currently,  there  is  a  strong  attempt  to  consolidate  the  many 
existing  specifications  on  every  conceivable  facet  of  reliability  into  a  very 
few  basic  triservice  and  NASA  coordinated  documents.  A  similar  consolida¬ 
tion  of  specifications  dealing  with  other  disciplines  encompassed  by  system 
effectiveness  will  mark  the  next  trend.  The  experience  of  one  company  in 
combining,  at  the  policy  level,  the  activities  of  reliability,  maintainability, 
human  engineering,  safety  engineering,  and  value  engineering  is  described. 
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